zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: About RagChecker  
GitHub labels: enhancement

Recently, I have noticed something similar to ragas, which is the [RagChecker](https://github.com/amazon-science/RAGChecker). It provides a new perspective to evaluate RAG pipelines which separately focuses on:
- Relevant chunk
- Irrelevant chunk used by the model
- Irrelevant chunk ignored by the model

This new perspective will provide a more detailed evaluation of the model’s performance, allowing for a deeper understanding of how different types of data chunks impact the evaluation process.

I am willing to design some new evaluation methods based on it. Please let me know if you are open to this idea, and I can provide further assistance or code examples.




@binghangli378 that is a very interesting Idea, reopening this to track more. Would you still like to help out on this?

@shahules786 something we can consider for #1010 ?

@binghangli378 Yes, this is very interesting. 
From what I observed they have a few more metrics that are not available in Ragas (note I just added #1174), I think the two metrics that would be beneficial are
1) self-knowledge: this would be something like a 1 - faithfulness score. Uses to measure how much of the generated response contains knowledge from LLM itself. 
2) noise sensitivity: this is more interesting, I think what they are trying to achieve is 
```
number of incorrect claims in the generated answer that came from irreverent chunks / total number of claims in the answer
```
This could be used to understand how bad noise in the context is affecting the quality of the generated answer. 
I also found this [paper](https://arxiv.org/pdf/2401.14887) showing noise in retrieved-context effects answer quality.
 
tagging you guys in case if you're interested in contributing. I have added them to the metrics roadmap. 
@sky-2002 @vaishakhRaveendran  

I can take up `noise-sensitivity` In fact, we had discussed something similar what I was referring to as attributing each claim in answer to some context.

@sky-2002 Sure, Can you please comment in this [issue](https://github.com/explodinggradients/ragas/issues/1185) so that I can assign it to you?


Closing as completed.
Repository: explodinggradients/ragas
Issue: Typo in results key -> context_ relevancy
Root cause: Validation Gap

`results.keys()` return

`dict_keys(['answer_relevancy', 'context_ relevancy', 'faithfulness', 'ragas_score'])`

where there is a whitespace in `context_ relevancy`

Due to https://github.com/explodinggradients/ragas/blob/9f54d016fd0743c4e6abac585cba76d36e3b1436/src/ragas/metrics/context_relevance.py#L110

Thanks a lot for pointing this out @ogencoglu, I've fixed it and will be out in the next release

Apologies for the delay from my end but do let us know your feedback on using ragas :)
Repository: explodinggradients/ragas
Issue: Buggy tutorials
Root cause: Documentation Deficiency

[ X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
A clear and concise description of what the bug is.

This tutorial doesn't seem to be functioning properly. I question whether the developer has actually run this code. Even the class names aren't accurate, like 'OpenAIEmbedding' for instance.

Among others: 

https://docs.ragas.io/en/latest/howtos/applications/compare_embeddings.html

It would have been beneficial to provide a sample file with the necessary libraries for execution beforehand.


Ragas version:
Python version:

**Code to Reproduce**
Share code to reproduce the issue

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Greeting -->
Hello @amr-cloudforce! I'm here to help you with any bugs, questions, or contributions you might have. Let's work together to resolve the issue you're facing.

<!-- Answer -->
The tutorial on comparing and choosing embeddings for data using the Ragas library does not function properly because the class names like `OpenAIEmbedding` are inaccurate. The correct class name should be `OpenAIEmbeddings`.

Incorrect usage in the tutorial:
```python
openai_model = OpenAIEmbedding()
```

Correct usage should be:
```python
openai_model = OpenAIEmbeddings()
```

Additionally, the Ragas library uses a wrapper class `LangchainEmbeddingsWrapper` to handle embeddings, which should be instantiated correctly. Here is an example of how to properly set up the embeddings using the Ragas library:

```python
from ragas.embeddings.base import LangchainEmbeddingsWrapper
from langchain_openai.embeddings import OpenAIEmbeddings

openai_model = OpenAIEmbeddings(model="text-embedding-ada-002")
wrapped_model = LangchainEmbeddingsWrapper(openai_model)
```

This ensures that the embeddings are correctly instantiated and wrapped for use within the Ragas framework <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/applications/compare_embeddings.md)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/embeddings/base.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@amr-cloudforce thanks for pointing it out! would you like to take a hand at contributing a fix? would really appreciate the help and I can help you out as well 🙂
Repository: explodinggradients/ragas
Issue: Can we turn off eval verbosity
Root cause: Unimplemented Feature Gap

Can we prevent printing the progress and other logs?

now right now but I will make a fix for this by next week.

but could you tell me a bit more about what logs that you wanted to avoid? would it show no logs and only the results?

A `verbose` or `verbosity` option to turn on/off all printing would be great

@ogencoglu  As a workaround, you can do something like this:
https://github.com/repllabs/fastrepl/blob/cf4e4f6ee743691d1df2d5dadf0f61add3ee95b0/fastrepl/eval/model/ragas.py#L194

I am integrating ragas into different platform and below messages are annoying. Can we turn it off in settings please?

![image](https://github.com/explodinggradients/ragas/assets/7220691/ebe425d6-5ea7-452f-aa1d-a6684bd805b1)


hey @atursunov - could you explain a bit more about why you want *every* logs removed so that I can understand your usecase a bit better? 
without progress bars it will be hard to know the progress for `evaluations()`

It would be nice if there are couple of verbosity options. Standard scikit-learn style stuff.

I see that you feel very strongly about this :)
have you had a chance to take a look at the latest? We have reduced the logs to just a progress bar as shown.
![image](https://github.com/explodinggradients/ragas/assets/5261489/320d72a8-e13a-4cf8-8327-0ea5187ffc26)
do you want to disable that too?

the reason is that I'm also thinking about how to show debug logs. the 2 options we have are `logging` module in python, which is not so great for LLM outputs and such and the `callbacks` in langchain_core which are very good at tracing. docs here [Logging and tracing | Ragas](https://docs.ragas.io/en/latest/howtos/applications/tracing.html)

would love to here you thoughts on this

@jjmachan  I'll open up another issue for this, but I was curious if there was an option to do the exact opposite, more verbose logs. I would be curious to understand ragas is arriving at the scores it does. 

Currently for example, on faithfulness, I can add a print statement here to see the rough json output of the scores:
https://github.com/explodinggradients/ragas/blob/30c830377644dce94e410f3ce3a8aa4339bb74a9/src/ragas/metrics/_faithfulness.py#L201

But it would be nice if I could turn that on with a config flag or something. Does this already exist?

hey @alexjyong where were you able to figure it out?
the recommended way is to use some traces

closing this for now, feel free to reopen if needed

@jjmachan  I'm sorry I have been working on other projects at work so I haven't had much time to play with ragas. I'll check out the latest features and see if that fits our needs. thank you!
Repository: explodinggradients/ragas
Issue: Exception handling
Root cause: Validation Gap

[ x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
https://github.com/explodinggradients/ragas/blob/0172bdaa13202501237d8525111473f85ecc92a9/src/ragas/testset/docstore.py#L253-L254
based on the current implementation, if result returned is empty for any doc, it will raise an error and break the whole process. This should not be the right behavior since we cannot try catch the error and all processed data will be lost.

Should change this to warning.
Repository: explodinggradients/ragas
Issue: [R-278] Battle test experimental synthetic data generation
Root cause: Unimplemented Feature Gap

- [X] Test generation from blogs
  - [X] resolve errors associated with headlines splitter
- [X] Implement hierarchical splitter
- [X] Experiment with persona wise generation
- [ ] Add next and prev relationships to nodes

<sub>From [SyncLinear.com](https://synclinear.com) | [R-278](https://linear.app/exploding-gradients/issue/R-278/battle-test-experimental-synthetic-data-generation)</sub>
Repository: explodinggradients/ragas
Issue: Bug in context_relevancy _compute_score method
Root cause: Validation Gap

I was debugging the code and found that if llm return "Insufficient Information" then tokenize on llm response is not correct as llm response contains no dot at the end and  comparison literal has dot at the end. 
See below screenshot of comparision
<img width="604" alt="image" src="https://github.com/user-attachments/assets/4c4fef7c-e882-4ba1-a8f7-8881ed384253">

Prompt instruction :
    instruction="""Please extract relevant sentences from the provided context that is absolutely required answer the following question. If no relevant sentences are found, or if you believe the question cannot be answered from the given context, return the phrase **"Insufficient Information"**.  While extracting candidate sentences you're not allowed to make any changes to sentences from given context.""",




<!-- Greeting -->
Hey there, @salman-khandu! I'm here to help you out with any bugs, questions, or contributions you have in mind. Let's get started!

<!-- Answer -->
The issue you're encountering is due to a mismatch in the literal comparison of the "Insufficient Information" response. The check in the `_compute_score` method expects a period at the end of the response, but the LLM response does not include it.

To fix this, you can standardize the comparison by stripping any trailing punctuation from the response before performing the check. Here is a modified version of the `_compute_score` method:

```python
def _compute_score(self, response: str, row: t.Dict) -> float:
    context = "\n".join(row["contexts"])
    context_sents = sent_tokenize(context)
    standardized_response = response.strip().rstrip('.')
    indices = (
        sent_tokenize(standardized_response)
        if standardized_response.lower() != "insufficient information"
        else []
    )
    if len(context_sents) == 0:
        return 0
    else:
        return min(len(indices) / len(context_sents), 1)
```

This change ensures that the comparison is consistent regardless of whether the LLM response includes a period at the end. The `rstrip('.')` method removes any trailing periods from the response before the comparison is made.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


thanks for reporting this! will be improve with #1012 

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Do we need to chunk documents before text set generation?
Root cause: Documentation Deficiency

The embedding model is used for TestsetGenerator:
```py
generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embedding_model)
dataset = generator.generate_with_langchain_docs(documents, test_size=100, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
```
Does this mean we have to chunk the documents to make documents shorter than the max_seq_length of the embedding model?

<!-- Greeting -->
@hanfei1986 Hello there! I'm here to help you with any bugs, questions, or contributions while we wait for a human maintainer. Let's squash those bugs together! 🤖

<!-- Answer -->
Yes, you need to chunk documents before text set generation when using the embedding model for `TestsetGenerator`. This ensures that the documents do not exceed the `max_seq_length` of the embedding model. The `TokenTextSplitter` is typically used for this purpose:

```python
from langchain.text_splitter import TokenTextSplitter

splitter = TokenTextSplitter(chunk_size=1000, chunk_overlap=100)
```

This splitter will chunk the documents into smaller pieces before they are processed and stored in the `InMemoryDocumentStore` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/generator.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/customisations/aws-bedrock.ipynb)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/tests/unit/testset_generator/test_document.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@hanfei1986 @dosu was wrong here, don't chunk the documents. We do some chunking for the documents ourselves in a way that makes sense for the testset generation module

<!-- Answer -->
You do not need to chunk the documents before text set generation. The `TestsetGenerator` module handles the chunking of documents internally in a way that makes sense for the test set generation process. This means you can pass your documents directly to the `generate_with_langchain_docs` method without worrying about their length relative to the `max_seq_length` of the embedding model.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: BaseLLM issue with ContextRelevancy and AnswerRelevancy
Root cause: Validation Gap

Hello! Thanks for the great work :) 

I've run into a bug while trying to use a local LLM. I cannot compute either `ContextRelevancy` or `AnswerRelevancy` when using a langchain BaseLLM, due to this Exception in line 36 in `ragas.metrics.llms`:

```
20      def generate(
21          prompts: list[ChatPromptTemplate],
22          llm: BaseLLM | BaseChatModel,
23          n: t.Optional[int] = None,
24          temperature: float = 0,
25          callbacks: t.Optional[Callbacks] = None,
26      ) -> LLMResult:
27          old_n = None
28          n_swapped = False
29          llm.temperature = temperature
30          if n is not None:
31               if isinstance(llm, OpenAI) or isinstance(llm, ChatOpenAI):
32                    old_n = llm.n
33                    llm.n = n
34                    n_swapped = True
35               else:
36  --->           raise Exception(
37                     f"n={n} was passed to generate but the LLM {llm} does not support it."
38                     " Raise an issue if you want support for {llm}."
39                )
```

The issue arises because when `ContextRelevancy` and `AnswerRelevancy` call this function they pass in `n=self.strictness`, eg in `ragas.metrics.answer_relevance`:

```
75        results = generate(
76            prompts,
77            self.llm,
78            n=self.strictness,
79            temperature=self.temperature,
80            callbacks=batch_group,
81        )
```

However `strictness` must be an integer due to both class's post_init:
```
54    def __post_init__(self: t.Self):
55        self.temperature = 0.2 if self.strictness > 0 else 0
```

Passing `strictness=None` would resolve the Exception issue but yields another error due to the post_init integer comparison.

Is there any scope to change the Exception to allow for n=0 or similar (eg change line 30 in `ragas.metrics.llms` to `if n is not None and n>0`?

If not, we are currently unable to use these metrics with a non-OpenAI LLM as far as I can tell.

Thank you!


Thank you for pointing this out @arikupe, sadly This is a current limitation of ragas ATM. I'm working on a fix and will merge it by next week. I'll reach out to you right after that fix has been released.

I hope you can wait till  the mean time

Hi @jjmachan, thanks so much for looking into this!

Of course, I can use OpenAI for now. Thanks again :)

Thanks a lot for this great repo. I see the same issue with ContextRecall as well. This fix will be great to have whenever it is released.

this has been fixed with #197 and will be released shortly, please do try it out and reopen the issue if its still a problem :)
Repository: explodinggradients/ragas
Issue: Bug - "key" argument not used in "get_or_init" 
Root cause: Validation Gap

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
In the `ragas.utils.get_or_init`, the `key` argument is unused, causing other functions to not work.

The function is:
```
def get_or_init(
    dictionary: t.Dict[str, t.Any], key: str, default: t.Callable[[], t.Any]
) -> t.Any:
    _value = dictionary.get("key")
    value = _value if _value is not None else default()

    return value
```

It should be:
```
def get_or_init(
    dictionary: t.Dict[str, t.Any], key: str, default: t.Callable[[], t.Any]
) -> t.Any:
    _value = dictionary.get(key)
    value = _value if _value is not None else default()

    return value
```

Ragas version: 0.1.10
Python version: 3.11.9

**Code to Reproduce**
```
from langchain_openai import ChatOpenAI
from ragas.integrations.langchain import EvaluatorChain
from ragas.metrics import faithfulness

llm = ChatOpenAI(model='gpt-4o')
print(llm.model_name)
faithfulness_chain = EvaluatorChain(metric=faithfulness, llm=llm)
print(faithfulness_chain.metric.llm.langchain_llm.model_name)
```
Output:
```
gpt-4o
gpt-3.5-turbo
```


<!-- Greeting -->
Hey @abramowskia! 👋 I'm here to help you with any bugs, questions, or contributions you have in mind. Let's squash those bugs together!

<!-- Answer -->
The bug in the `ragas.utils.get_or_init` function is due to the incorrect usage of the `key` argument. The current implementation incorrectly uses the string `"key"` instead of the variable `key`.

The current implementation:
```python
def get_or_init(
    dictionary: t.Dict[str, t.Any], key: str, default: t.Callable[[], t.Any]
) -> t.Any:
    _value = dictionary.get("key")
    value = _value if _value is not None else default()

    return value
```

The corrected implementation should be:
```python
def get_or_init(
    dictionary: t.Dict[str, t.Any], key: str, default: t.Callable[[], t.Any]
) -> t.Any:
    _value = dictionary.get(key)
    value = _value if _value is not None else default()

    return value
```

This change ensures that the `key` argument is properly used to retrieve the value from the dictionary <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/.github/ISSUE_TEMPLATE/bug_report.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


fix has been merged in 🙂
Repository: explodinggradients/ragas
Issue: Any available dataset?
Root cause: Documentation Deficiency

When test our ragas, I found the dataset created by gpt-4 may be not that ideal. Any recommend dataset avalible? 


<!-- Answer -->
You can use the `explodinggradients/prompt-engineering-guide-papers` dataset from Hugging Face to prepare your test dataset. This dataset contains questions and ground truth answers, which you can use to evaluate your RAG pipeline. Here is how you can load and use this dataset:

```python
from datasets import load_dataset

eval_dataset = load_dataset("explodinggradients/prompt-engineering-guide-papers")
eval_dataset = eval_dataset["test"].to_pandas()
eval_dataset.head()
```

Additionally, for evaluation, you can use the Amnesty QA dataset from Hugging Face, which contains questions, contexts, and ground truth answers:

```python
from datasets import load_dataset
# loading the V2 dataset
amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")
amnesty_qa
```

These datasets should help you generate a comprehensive test dataset for evaluating your RAG system <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/prepare_data.ipynb)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/evaluation.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I'm hoping this helped? feel free to reopen this if not
Repository: explodinggradients/ragas
Issue: Can't run evaluate() from llamaindex integration
Root cause: Interface Contract Mismatch

Hello team 👋

When I try to reproduce your [llamaindex notebook](https://github.com/explodinggradients/ragas/blob/main/docs/integrations/llamaindex.ipynb), without modifying anything, I've got an error with:

`result = evaluate(query_engine, metrics, eval_questions, eval_answers)`

It says:

`TypeError: evaluate() takes 3 positional arguments but 4 were given`

Any idea on how to make it work? Thanks!

Hey there @ulyssebottello, I really appreciate the time you took to raise this issue and my sincere apologies for the delay, I was on leave last week but I've just released v0.0.12 which has fixed this issue for you.

I would really appreciate it if you take a look and let me know if it works at your end as well :)



@jjmachan 
I have the new version ragas 0.0.12 and I'm getting the same error.

result = evaluate(query_engine, metrics, eval_questions, eval_answers)
TypeError: evaluate() takes from 1 to 3 positional arguments but 4 were given

ohh that is strange because it seems to work at my end, @ryyeung Could you share the entire stack trace, would really help figure out what is wrong

I'm using the Llama Index Github Data Loader. Could that cause problems? @jjmachan 

![Screenshot 2023-09-06 at 9 05 29 PM](https://github.com/explodinggradients/ragas/assets/97981608/4deff662-7054-4407-9230-897ed3565587)


@jjmachan Any updates?

apologies @ryyeung was not able to reproduce at my end. are you on discord? if yes can you give me your username or email id? we can do a bit more debugging there and try and fix it.

@jjmachan No worries! Yes, I just joined. My username is: ryyeung

closing this after having a wonderful debug session with @ryyeung!

Well done guys! @jjmachan @ryyeung 🤝

I am having the same issue, how did you solve it? @jjmachan @ryyeung

I am running:
ragas==0.021
openai==1.3.5
llama-index==0.9.8.post1
langchain==0.0.341