zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: how to use RAGAS metrics for GPT3.5 Finetuned model
Root cause: Documentation Deficiency

Hi all,
I understand how RAGAS work for RAG system.
I have use case where I have fine-tuned a GPT3.5 model on my data and using this model for question-answering.
I want to know how if I can use RAGAS to evaluate this fine-tuned model, as it does not have context/chunks to be passed into RAGAS metrics function.
Can anyone help me with "how to use RAGAS metrics for my GPT3.5 Finetuned model"

Hey @s-kumar17 thanks for raising the question. You could use the `faithfulness` and `answer_relevancy` metric here. Please refer to this to see it in action https://gpt-index.readthedocs.io/en/stable/examples/finetuning/openai_fine_tuning.html#evaluation
Repository: explodinggradients/ragas
Issue: Docs page is not found
GitHub labels: bug

documentation

https://docs.ragas.io/en/latest/howtos/applications/compare_embeddings.html shows 404


<!-- Greeting -->
Hey @rodion-m! I'm here to help you with any bugs, questions, or contributions while we wait for a human maintainer. If you need assistance, feel free to ask!

<!-- Answer -->
The link to the documentation page for comparing embeddings shows a 404 error because the file `howtos/applications/compare_embeddings.md` is missing from the `mkdocs.yml` configuration. Ensure that the file `howtos/applications/compare_embeddings.md` exists in the specified directory and is correctly referenced in the `mkdocs.yml` file <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/mkdocs.yml)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


> https://docs.ragas.io/en/latest/howtos/applications/compare_embeddings.html shows 404

this could be the most relevant one 
https://docs.ragas.io/en/stable/howtos/applications/compare_embeddings.html

hey guys - thanks for reportings these
the main docs page has a few missing links since we revamped it for v0.2 😅 

will clean up all of these before the release 🙌🏽 

the `.html` is dropped on newer docs. The page in question is now at https://docs.ragas.io/en/latest/howtos/applications/compare_embeddings
Repository: explodinggradients/ragas
Issue: Testset Generation: bringing continual learning to RAG pipelines
Root cause: Unimplemented Feature Gap

We started ragas with `ground-truth`free evaluations so that you didn't have to put significant upfront effort into building an ideal test set before running evaluations. **Creating a test set needs substantial upfront investment in time, money, human hours and expertise to get it right.** It is also a continuous process as your product and ML model evolve to cater to diverse use cases.  We are exploring the possibilities of synthetic test set generation because

1. As RAG users get more mature and go into production, having a solid test set and evaluation strategies becomes critical to give users a seamless experience. This means they have to put more time into building solid test sets and evaluation methodologies.
2. ground-truth free evaluation has its limitations. It is very effective in quantifying aspects like faithfulness but **ground-truth free evaluation cannot be used to ensure aspects like answer correctness which is also very important**. Here a synthetic test set with ground-truth can be of high utility.

The whole focus of the Ragas library is to help you build more reliable RAG applications which is why with the next leg of Ragas we'll be focusing a lot more on test set generation and continual learning of RAG pipelines. The goal is to leverage custom LLMs and Data-Centric AI techniques to

1. Build more robust paradigms for test set generation. 
    1. Many libraries already have some sort of test set generation but they have a few shortcomings. Ideally, the test set should have a good distribution of easy -> hard questions across different tasks/situations as seen in production.
2. Tools to scale up and reduce the cost of test set generation. 
    1. Works like self-instruct and evol-instruct have proven that LLMs can generate human-quality synthetic data. **We are working on paradigms to generate high-quality synthetic data generation specific to RAG**. Ref [[1]](https://arxiv.org/abs/2306.08568) [[2]](https://arxiv.org/abs/2212.10560)
3. Methodologies to continuously add to and improve the test set as your RAG pipelines evolve using other data points like logs and feedback.

there is a lot of work to be done but with the v0.1 release of Ragas, we'll be releasing features in this direction. In the meantime, we would love to hear your opinions, expectations, suggestions and ideas about this too :)

Team Ragas
Repository: explodinggradients/ragas
Issue: Request to replace fuzzywuzzy with rapidfuzz in the Jaccard(Similarity) class due to licensing restrictions
Root cause: External Dependency Breakage

I would like to request that ragas uses rapidfuzz instead of fuzzywuzzy in the Jaccard(Similarity) class. fuzzywuzzy uses the GPL-2.0 license, which prevents its usage in projects with more restrictive licensing requirements. 

rapidfuzz offers similar functionality but is licensed under the more permissive MIT license, which would allow broader adoption and remove the licensing conflict.

In our project, we cannot use any libraries under GPL licenses, including fuzzywuzzy. As ragas currently depends on fuzzywuzzy, this creates a conflict. Replacing it with rapidfuzz would enable us to continue using ragas while staying within our project's licensing guidelines.

If there is any alternative approach to solve this issue, I would appreciate your suggestions. We want to ensure compatibility with more permissive licenses like MIT.

Thank you for your time and support!


Hey @AishAmalgam thanks for pointing it out. It's use in metrics was already removed but the dependency remained due to one function in experimental testgen. Just raised a PR to replace it.

resolved already 🙂
Repository: explodinggradients/ragas
Issue: give me the conclusion for this topic we are getting far from the real context.

              give me the conclusion for this topic we are getting far from the real context.
I know it is better to let the ragas do the embeddings but if i want to consider the cost or price for this
is it better to use my own  calculated embeddings and then in some ways or method make sure that my embeddings are suitable or compatible for using ragas,

_Originally posted by @amin-kh96 in https://github.com/explodinggradients/ragas/issues/1358#issuecomment-2376449771_
            

this is a duplicated
Repository: explodinggradients/ragas
Issue: Get explanations along with metric scores
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

I want to get all the intermediate steps performed while getting a metric score. This will be helpful to understand why the model took a particular action. 


he @saiprasath21 you can do that with [Explainability through Logging and tracing - Ragas](https://ragas--1388.org.readthedocs.build/en/1388/howtos/applications/tracing/?h=trac) or writting your own langchain callbacks

but can you check out the docs and let me know if it works for you 🙂 

Hey @jjmachan , yes this helps a lot. 

I'm now passing a langfuse call handler to ragas using the evaluate func. However, all my dataset item spans are recorded under the same trace. Is there a way to create a separate trace for each dataset item?
Repository: explodinggradients/ragas
Issue: Quickstart example faithfulness result showing nan
Root cause: Validation Gap

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I just copied and paste quick start example code on my end everything is good except faithfulness it shows nan.
Is this normal?


**Code Examples** 

from datasets import Dataset 
import os
from ragas import evaluate
from ragas.metrics import faithfulness, answer_correctness

os.environ["OPENAI_API_KEY"] = "your-openai-key"

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
    'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}

dataset = Dataset.from_dict(data_samples)

score = evaluate(dataset,metrics=[faithfulness,answer_correctness])
score.to_pandas()

**Additional context**
I'm using ragas 0.1.20 version


<!-- Answer -->
The issue of the faithfulness metric showing 'nan' in the quickstart example is likely due to no statements being generated from the answer. This can happen if the number of statements (`num_statements`) is zero, which results in the score being set to `np.nan` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_faithfulness.py)</sup>. 

To resolve this, ensure that the answer is properly segmented into statements. You might need to check the sentence segmentation process to ensure it is correctly breaking down the response into simpler statements.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: Possible bug? context_<space>relevancy -> has space in between
Root cause: Validation Gap

Hi team, 

I'm new with RAGAS, however I found this : 
```

from ragas.metrics import faithfulness, answer_relevancy, context_relevancy, context_recall
from ragas.langchain import RagasEvaluatorChain

# make eval chains
eval_chains = {
    m.name: RagasEvaluatorChain(metric=m) 
    for m in [faithfulness, answer_relevancy, context_relevancy, context_recall]
}

for name, eval_chain in eval_chains.items():
    score_name = f"{name}_score"
    print(f"{score_name}: {eval_chain(result)[score_name]}")

```

using template code, i found that the score name for context_ relevancy -> it has space in between. Is it just me? 
![image](https://github.com/explodinggradients/ragas/assets/11275004/22d76ffc-7dae-4d6f-9ac8-6c353b0c383d)
(dont mind the len=3, i was using different code)

Hey @stepkurniawan this definitely seems like a bug and I will be fixing this shortly

could you also check the context_recall and context_precision please? :)

@stepkurniawan which version or ragas are you using, I had fixed this previously and am not able to reproduce this now. Can you try updating ragas to the latest version and try again?

<!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
.MsoChpDefault
	{mso-style-type:export-only;}
@page WordSection1
	{size:612.0pt 792.0pt;
	margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
	{page:WordSection1;}
-->Sorry, could you tell me how to check the Version and update the Version?  Best regards, Stephen Kurniawan Von: Jithin JamesGesendet: Wednesday, 20 September 2023 15:57An: explodinggradients/ragasCc: Stephen Kurniawan; MentionBetreff: Re: [explodinggradients/ragas] Possible bug? context_<space>relevancy -> has space in between (Issue ***@***.*** which version or ragas are you using, I had fixed this previously and am not able to reproduce this now. Can you try updating ragas to the latest version and try again?—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***> 

```python
import ragas
ragas.__version__
```

and you can update it by running
```bash
pip install -U ragas
```

version should be 0.0.13 - 0.0.14

closing this hoping that fixed it @stepkurniawan, if you are still running into the issue please feel free to reopen this, more than happy to help fix it :)
Repository: explodinggradients/ragas
Issue: An 'answer_relevancy' error occurred.
Root cause: Unimplemented Feature Gap

- [x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
The following error occurred
NotImplementedError: adapt() is not implemented for answer_relevancy metric

**Code Examples**
```
from ragas.llms import llm_factory
from ragas import evaluate
from datasets import Dataset
from ragas.metrics._answer_relevance import AnswerRelevancy
from ragas.metrics._answer_relevance import ResponseRelevancy

# Ragasの評価用LLMの設定
LLM_NAME = "gpt-4o-mini"
ragas_llm = llm_factory(model=LLM_NAME)

# Adapt the ResponseRelevancy metric to Japanese
answer_relevancy = ResponseRelevancy()
answer_relevancy.adapt(language="japanese")

dataset = Dataset.from_dict(
    {
        "question": questions,
        "answer": answers,
        "contexts": contexts,
        "ground_truth": ground_truths,
    }
)

score = evaluate(
    dataset,
    llm=ragas_llm,
    metrics=[
        answer_relevancy,
    ],
)
print(score)
score.to_pandas()
```

<!-- Answer -->
I couldn't find specific information about the implementation status of the `adapt` method in the `AnswerRelevancy` class within the Ragas framework. However, the error message you're encountering, `NotImplementedError: adapt() is not implemented for answer_relevancy metric`, suggests that the `adapt` method is not yet implemented for the `AnswerRelevancy` metric.

You might want to check if there are any updates or recent changes in the repository that address this issue. Additionally, you could consider using other metrics that support the `adapt` method or explore alternative approaches to achieve your goal.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Try with metrics collections `from ragas.metrics.collections import AnswerRelevancy`
Repository: explodinggradients/ragas
Issue: I can't generate Test Data.
Root cause: Validation Gap

----> 1 testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25},raise_exceptions=False)

2 frames
/usr/local/lib/python3.10/dist-packages/ragas/testset/docstore.py in add_nodes(self, nodes, show_progress)
    251         results = executor.results()
    252         if not results:
--> 253             raise ExceptionInRunner()
    254 
    255         for i, n in enumerate(nodes):

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.


<!-- Greeting -->
Hey @nandhu-nr! I'm here to help you with that issue. Let's get to the bottom of it together.

<!-- Answer -->
The `ExceptionInRunner` error occurs when an exception is raised in the executor, specifically when the results from the executor are empty, indicating that the runner thread encountered an issue. To debug this, you should review the traceback provided in the error message. Additionally, since you have set `raise_exceptions=False`, it should only show a warning message instead of raising the exception, but it seems like the exception is still being raised.

You might want to check if there are any issues with the input documents or the configuration of the `generate_with_langchain_docs` function. Ensure that the `documents` parameter is correctly formatted and that all other parameters are set as expected. If the problem persists, consider checking for any updates or patches to the Ragas library that might address this issue.

For similar issues, you can refer to the closed issue [Testset generation. ValueError: a cannot be empty unless no samples are taken](https://github.com/explodinggradients/ragas/issues/625), which was resolved by a pull request <sup>[[1]](https://github.com/explodinggradients/ragas/issues/625)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


imagine using bot to answer this kind of questions
Repository: explodinggradients/ragas
Issue: Issue with Context type
Root cause: Documentation Deficiency

I am trying to run evaluate on outputs generated by GPT4. I have the columns structured in the desired format however am running into the following error

Dataset feature "contexts" should be of type Sequence[string[, got <class 'datasets.features.features.Value'>

Any tips on how to resolve? Thank you!

Hi @netranga, the context should be given as a list of strings. Can you make sure that this is correct? 

Yes this is the correct input - now hitting the following error:
"Must provide an 'engine; or 'deployment_id' parameter to create a <class 'openai.api_resources.chat_completion.ChatCompletion'>.

I have set my API Key to my azure_token - I am accessing models through Azure but running my code with Sagemaker. Would this be causing an issue? 

hey @netranga sorry about the late response but just published a new guide [here](https://github.com/explodinggradients/ragas/blob/main/docs/guides/quickstart-azure-openai.ipynb) that should help with this issue. Hope you find it useful

Hi @jjmachan im trying to access this ../quickstart-azure-openai.ipynb/ it is no longer available
Repository: explodinggradients/ragas
Issue: Test dataset Issue
Root cause: Validation Gap

WARNING:ragas.testset.docstore:Filename and doc_id are the same for all nodes.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-40-b2341dc57254>](https://localhost:8080/#) in <cell line: 11>()
      9 
     10 # generate testset
---> 11 testset1 = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}, raise_exceptions=False)

3 frames
[/usr/local/lib/python3.10/dist-packages/ragas/testset/docstore.py](https://localhost:8080/#) in get_random_nodes(self, k, alpha)
    325         prob = prob / np.sum(prob)
    326 
--> 327         nodes = self.run_config.rng.choice(
    328             np.array(self.nodes), size=k, p=prob
    329         ).tolist()

numpy/random/_generator.pyx in numpy.random._generator.Generator.choice()

ValueError: a cannot be empty unless no samples are taken

<!-- Answer -->
The error you're encountering, `ValueError: a cannot be empty unless no samples are taken`, is a known issue that occurs when the `sample` function is called with an empty sequence. This can happen if the `test_size` is greater than the number of available nodes in the `docstore`.

To resolve this issue, ensure that the `docstore` contains enough nodes to match the `test_size`. You can add a check to handle cases where there are not enough nodes:

```python
current_nodes = [
    CurrentNodes(root_node=n, nodes=[n])
    for n in self.docstore.get_random_nodes(k=test_size)
]

if len(current_nodes) < test_size:
    raise ValueError("Not enough nodes in the docstore to generate the test set.")
```

Additionally, make sure that the `documents` list is not empty before passing it to the `TestsetGenerator`. You can add a check to verify that documents are loaded successfully:

```python
documents = SimpleDirectoryReader(CONFIG.file_path).load_data()

# Check if documents are loaded
if not documents:
    raise ValueError("No documents were loaded. Please check your file path or parameters.")

print("load file num:", len(documents))
```

This will ensure that you do not attempt to generate a test set with an empty list of documents <sup>[[1]](https://github.com/explodinggradients/ragas/issues/1194)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: answer_relevancy score decreased drasctically 
Root cause: External Dependency Breakage

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
The code is identical only difference is the packages version ex) ragas, langchain it answer_relevancy used to be above 0.8 now it is somewhere around 0.7 I have no idea what is the issue.


<!-- Answer -->
I found a similar open issue titled [Bug in Answer Similarity Calculation](https://github.com/explodinggradients/ragas/issues/506), which might be related to your problem. This issue is currently closed and was last updated on June 01, 2024 <sup>[[1]](https://github.com/explodinggradients/ragas/issues/506)</sup>.

Regarding the decrease in the `answer_relevancy` score, it's important to consider the dependencies and their versions, as they can influence the score. However, I couldn't find specific information on changes in recent versions of the `ragas` package that could affect the `answer_relevancy` score <sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_relevance.py)</sup>.

One parameter that could impact the score is the `strictness` parameter in the `ResponseRelevancy` class. It determines the number of questions generated per answer, which can affect the score calculation <sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_relevance.py)</sup>.

If you haven't already, you might want to check the versions of the `ragas` and `langchain` packages you're using, as changes in these packages could potentially affect the score.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


what was the version before parameter change and after? I'm using ragas 0.1.13 version

<!-- Answer -->
I wasn't able to find specific information on the version changes related to the `strictness` parameter or any other parameter changes in the `ragas` package that could affect the `answer_relevancy` score. You might want to check the release notes or changelogs of the `ragas` package for detailed information on version changes.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: running quickstart.ipynb fails
Root cause: Environment Incompatibility

Hi,

When running the quickstart.ipynb notebook, i got the following errors:

- missing tiktoken (fixed by adding !pip install tiktoken)
- missing langcahin embedding (fixed by !pip install langchain==0.0.288)


Thank you for bringing this up @wzeyal and suggesting the fixes. I will take a look at this and make sure the dependencies are updated :)

Thanks !