zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Output testset when using Llama 3 8B instruct model is not proper.
Root cause: Validation Gap

The testset that I generated using llama 3 8B instruct model has problematic output. Unnecessary phrases are there in the output alongside the questions. I have given an example below.
  0: 'Here is a question that can be fully answered from the given context using the
    keyphrase "Employee Self Service":


    What is the procedure for an employee to apply for marriage leave through the
    Employee Self Service portal?'
  1: 'Here is a rewritten version of the question:


    "What''s the daily limit for foreign exchange expenses when booking travel?"


    I shortened the question by removing unnecessary words and used an abbreviation
    ("expenses" instead of "foreign exchange entitlements"). I also rephrased the
    question to make it more concise and indirect, while still conveying the same
    meaning.'
    
    How can i solve this? Can I adjust the prompts to the model? If yes, how?


hey @Nandakishore-Thekkadathu thank you for raising this issue 🙂 

this is a hard one to debug because llama 3 8B is a smaller model that doesn't have enough params to give useful results. Any chance you can use some other models? Like the bigger ones (gpt4, claude etc?)
Repository: explodinggradients/ragas
Issue: I've been encountering an issue where Ragas generates test cases from the initially uploaded documents, even after updating the documents and rerunning the process. It either uses the old documents or mixes them with the new ones.
Root cause: Resource Mishandling

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
what is unclear to you? What would you like to know?

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 

### Code

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

testset = generator.generate_with_langchain_docs(
    documents,
    test_size=3,
    distributions={
        'simple': 0.5,
        'reasoning': 0.25,
        'multi_context': 0.25
    }
)


I had this same issue but it got solved after dividing the whole code into main and worker scripts.

I'm also running into this issue
Repository: explodinggradients/ragas
Issue: TestsetGenerator.from_langchain Generating failed, randomly stuck at 0% to 80%.
Root cause: External Dependency Breakage

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
TestsetGenerator.from_langchain Generating failed, randomly stuck at 0% to 80% And I've been consuming tokens from the OpenAI API.

By the way, my embedded documents are in Chinese. Would this potentially affect this case?

I am a student who has just started programming, and my understanding of related knowledge is limited. I would be very grateful if there were experts who could understand my incomplete questions and help me.

```
Filename and doc_id are the same for all nodes.
Generating:  70%|██████████████████████████████████████████████████████▌                       | 7/10 [01:29<00:43, 14.33s/it]
```

**Code Examples**
```
from langchain_text_splitters import RecursiveCharacterTextSplitter
import os
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings


os.environ["OPENAI_API_KEY"] = "sk-xxx"

with open("RAGAS\output.md", encoding='utf-8') as f:
    state_of_the_union = f.read()

text_splitter = RecursiveCharacterTextSplitter(
    # Set a really small chunk size, just to show.
    chunk_size=512,
    chunk_overlap=128,
    length_function=len,
    is_separator_regex=False,
    separators=[
    "###"
    ]
)

documents = text_splitter.create_documents([state_of_the_union])
print(documents[0])

generator_llm = ChatOpenAI(model="gpt-3.5-turbo")
critic_llm = ChatOpenAI(model="gpt-3.5-turbo")
embeddings = OpenAIEmbeddings()

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)
from ragas.testset.evolutions import simple, reasoning, multi_context,conditional

generator.adapt(language="chinese",evolutions=[simple, multi_context, conditional, reasoning])
generator.save(evolutions=[simple, reasoning, multi_context,conditional])

testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25},  is_async=False)

testset.to_pandas()

testset.to_pandas().to_csv("RAGAS\output.csv", index=False)
```

**Additional context**
Anything else you want to share with us? 


卡住了，估计就是json的输出问题了

我看好像是編碼不支持 "~\miniconda3\envs\my_env\Lib\site-packages\ragas\llms\prompt.py" line 286, encoding="utf-8"

I also encountered the same problem, has your problem been solved?

hey @SugarMoonn sorry to hear that but we'll get this working for you
- but is caching working at your end?
- which version of ragas are you using?
- how many documents do you have?

Closing this as fixed in latest versions.
Repository: explodinggradients/ragas
Issue: JSON parser as association 
Root cause: Architectural Constraint

**Describe the Feature**

Currently, the output parser is hardcoded as JSON Parser. There is not even a way to customize the prompt, which works poorly with models like Claude. 

It would be much easier to work with RAGAS if it would be possible to just pass custom parser to places where it's used.


**Why is the feature important for you?**

Many 'nan' values in GT in generated dataset.


<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Closing this as resolved now.

Try using `llm_factory` and the new metrics collections.
Repository: explodinggradients/ragas
Issue: Why does Ragas require a llm model when evaluating on datasets with answers and questions?
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Why does Ragas require a llm model when evaluating on datasets with answers and questions?

I don't know why llm is needed when calculating indicators such as context_precision or context_recall for datasets with both question and ground_truth and answer and context

**Code Examples**
from ragas import evaluate
result = evaluate(
    test_dataset,
    metrics=[
        context_precision,
    ]
)

this code requires an error :
**ValidationError: 1 validation error for ChatOpenAI
__root__
  Did not find openai_api_key, please add an environment variable `OPENAI_API_KEY` which contains it, or pass `openai_api_key` as a named parameter. (type=value_error)**

**Additional context**
I wonder what the role of LLM and Embeddings models is when evaluating the dataset with answer and question and ground truth and context already.


we use LLMs for evaluations 2 overcome 2 shortcomings
1. in a generative system, the final answer need not be the same as the ground_truth. So you need a smart way to see if both are correct. A good example is the [Answer Correctness | Ragas](https://docs.ragas.io/en/stable/concepts/metrics/answer_correctness.html).
2. To augment data needed. For metrics like a retriever, building an annotated dataset is extremely hard, using LLMs augments that. See [Context Recall | Ragas](https://docs.ragas.io/en/stable/concepts/metrics/context_recall.html) calculations

Hope that answers your question 🙂 

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: What does ground_truth refer to?
Root cause: Documentation Deficiency

```
DatasetDict({
    baseline: Dataset({
        features: ['question', 'ground_truths', 'answer', 'contexts'],
        num_rows: 30
    })
})
```
I wanted to know if the ground truth refers to the **ground truth passages that need to be retrieved** or the the **final answer**
I was a bit confused as I saw
`ground_truths: list[list[str]] ` in the documntation.
Should it not be `ground_truths: list[str]` ?

right now ground truth can be 2 things. 
1. The answer that is considered the ground truth
2. The ideal retrieved contexts you want to get.

the ground_truth right now is `list[list[str]]` because it is much for flexible, you can use it to store both. We will be changing this in the upcoming v0.1 release to `reference_answer` and `reference_contexts`.

Hope this helps :)
I'll update the documentation before closing this issue in that case.

Okay this makes a lot of sense! thanks
Repository: explodinggradients/ragas
Issue: [R-264] Add more metrics with v0.2
Root cause: Unimplemented Feature Gap

Reference free

* Generation
* Summarisation
* Code summary
* Textual summary

With Reference

* Generation for data types
* Text
* answer correctness

Code
SQL

<sub>From [SyncLinear.com](https://synclinear.com) | [R-264](https://linear.app/exploding-gradients/issue/R-264/add-more-metrics-with-v02)</sub>
Repository: explodinggradients/ragas
Issue: [R-265] Ragas API and docker image
Root cause: Unimplemented Feature Gap

- [ ] make ragas metrics deployable as a server
- [ ] make testset generation interactive with an API

making a simple documentation site to showcase how this might work would be a great first step to get feedback.

<sub>From [SyncLinear.com](https://synclinear.com) | [R-265](https://linear.app/exploding-gradients/issue/R-265/ragas-api-and-docker-image)</sub>

https://github.com/explodinggradients/ragas/issues/1270 also refer

that was an error from the bot 😅
Repository: explodinggradients/ragas
Issue: [R-266] New Propsal for `Prompt`
Root cause: Unimplemented Feature Gap

Detailed Proposal: [PromptObject - v0.2](https://ragas.notion.site/PromptObject-v0-2-e7bbe8cc7fd94ac8868e51d6493520f1)

## User Requirements

1. Create Simple prompts easily
2. Typed I/O
3. Adaption - stress test, use the current issues to understand the limitations
4. Ability to remove reasoning and other types
5. change demonstraction of the fly
6. change decoding strategy (this is the responsibility of the LLM)
7. unique identifier name for each prompt → this should also change when prompt is modified. Why?
   1. let’s say user A adapts and saves ragas at v0.1
   2. after a while he uses ragas again but with adapted language - but this time we had changed the prompt then it causes issues.

<sub>From [SyncLinear.com](https://synclinear.com) | [R-266](https://linear.app/exploding-gradients/issue/R-266/new-interface-for-llms)</sub>

finished
Repository: explodinggradients/ragas
Issue: [R-270] Synthetic test data generation - v3
Root cause: Unimplemented Feature Gap

Highly programmable knowledge graph-based synthetic data generation grounded on documents.

PR raised #1024

<sub>From [SyncLinear.com](https://synclinear.com) | [R-270](https://linear.app/exploding-gradients/issue/R-270/synthetic-test-data-generation)</sub>
Repository: explodinggradients/ragas
Issue: Ragas installation induced pandas segmentation fault 
Root cause: Environment Incompatibility

- [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Once I install ragas via `pip install ragas` in my (otherwise clean) venv, I receive a segmentation fault when importing ragas or pandas. Prior to the ragas installation, I'm able to import pandas with no issues. This issue persists even when installing latest master of ragas. 

Ragas version: 0.19.0
Python version: 3.12.3 (also attempted on 3.11.8)

**Code to Reproduce**
Share code to reproduce the issue
In a clean venv with the above python version, 
```bash
pip install pandas
pip install ragas
```
In a python script/notebook:
```python
import pandas
```

**Error trace**
Segmentation fault: 11

**Expected behavior**
A clear and concise description of what you expected to happen.
Clean import. 


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Unfortunately, I'm unsure if this is a Ragas issue because if you check [pyproject.toml](https://github.com/explodinggradients/ragas/blob/main/pyproject.toml) file you can see that we don't directly depend on pandas, we only use the available installation. 

this might be something specific to your virtualenv sadly.

which virtualenv are you using?

Yeah, I did take a look at the toml. I've tried with multiple clean venvs, the first one was created using python3 in the vanilla way: `python -m venv /path/to/new/virtual/environment`. I then used conda to create a venv as well, where all I ran was `conda install pandas` and `pip install ragas`. When I did this I received the same error as specified in the issue. 

Since this is likely a platform/venv problem I'll close this issue out.
Repository: explodinggradients/ragas
Issue: RagasOutputParser exception handler should work synchronously when is_async=False
Root cause: Architectural Constraint

**Describe the Feature**

The `is_async` value chosen when calling the `evaluate` function should be used for the call to `generate` within `RagasoutputParser`.

```python
class RagasoutputParser(PydanticOutputParser):
    async def aparse(  # type: ignore
        self, result: str, prompt: PromptValue, llm: BaseRagasLLM, max_retries: int = 1
    ):
        try:
            output = super().parse(result)
        except OutputParserException:
            if max_retries != 0:
                p_value = FIX_OUTPUT_FORMAT.format(
                    prompt=prompt.to_string(), completion=result
                )
                output = await llm.generate(p_value)
                result = output.generations[0][0].text
                return await self.aparse(result, prompt, llm, max_retries - 1)
            else:
                logger.warning("Failed to parse output. Returning None.")
                return None
        return output
```
Currently in RagasoutputParser if parsing fail then an attempt is made to correct the JSON but it is not compatible with the ability to work synchronously if the llm does not support async because `is_async` is True by default in `llm.generate()`.


**Additional context**


Maybe #975 will solve this problem, if not I'd be happy to contribute if help is needed!

Thanks !



which LLM are you using @BaptisteBtx ?

but unfortunatly you are right. we will be removing `is_async` flag soon because we switched to only use async for LLM calls. 

is there anything I can do to help you in the mean time?

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: run_config missing 
Root cause: Validation Gap

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I got this error ``` TypeError: MetricWithLLM.init() missing 1 required positional argument: 'run_config' ``` when I was trying to calculate the ragas scores


Ragas version: 0.1.9
Python version: 3.11

**Code to Reproduce**
```py
from ragas import evaluate
from langchain_openai.chat_models import AzureChatOpenAI
from langchain_openai.embeddings import AzureOpenAIEmbeddings
from ragas.llms import LangchainLLMWrapper


from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    ContextRelevancy,
)

azure_model = AzureChatOpenAI(
     deployment_name="xxxxxx",
     model="xxxxxx",
     api_version ="xxxxxx",
     openai_api_type="azure"
)

ragas_azure_model =LangchainLLMWrapper(azure_model)

azure_embeddings = AzureOpenAIEmbeddings(
    deployment="xxxxxx",
    model = "xxxxxx",
    api_version ="xxxxxx",
     openai_api_type="azure"
)

metrics = [
    faithfulness,
    answer_relevancy,
    ContextRelevancy
]

for m in metrics:
    m.__setattr__('llm',ragas_azure_model)

questions = df.Query.values
rag_answers = df.Answers.values
contexts = df.Context.values



from datasets import Dataset

data = {
    "question": questions,
    "answer": rag_answers,
    "contexts": contexts
}


dataset = Dataset.from_dict(data)


result = evaluate(dataset=dataset,
                  metrics=metrics,
                  llm = ragas_azure_model,
                  embeddings = azure_embeddings)
```


**Error trace**
<img width="579" alt="image" src="https://github.com/explodinggradients/ragas/assets/30107613/274ac35f-8765-4eea-a2c4-18185676b88e">

<img width="1116" alt="image" src="https://github.com/explodinggradients/ragas/assets/30107613/6f943fc9-0e1b-4ccb-a34b-43cab8699e6f">



**Expected behavior**
For the code to calculate the ragas scores.





@Mtuthuko  did you resolve this error? I am also facing the exact problem

hey @Mtuthuko were you able to fix this?

it seems like a small bug in the code

```py
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    # ContextRelevancy, # old
    content_relavancy
)

metrics = [
    faithfulness,
    answer_relevancy,
    context_relevancy
]
```

this should fix it. let me know if its still an issue

@Sudhakar17 could you share your code snippet and error msg? I'll check it

Closing after 8 days of waiting for the additional info requested.
Repository: explodinggradients/ragas
Issue: [R-274] Improvements in v3 synthetic data generation
Root cause: Unimplemented Feature Gap

Issues and improvements tracking for v3 synthetic data generation

- [X] Implement return type as  Node and relationship objects when querying schema
- [X] QA diversification
  - [X] Node weighing like testgen v2
  - [X] seeding and dedup mechanism to avoid creation of duplication QA pairs
- [X] Extractors
  - [ ] ~~~add extract_node to enable running extractor.extract_node(node) type~~~
  - [X] Merging extractors
    - [X] check input key values are the same as merge criteria
    - [X] output list of extractors - \[extractor_merged, unmerged_extractor\]
  - [ ] Explore function calling type extractors (low priority)
- [ ] Document splitter
  - [ ] Fix abstraction leaks
- [ ] prompts
  - [ ] rename prompt variable names
- [X] Implement Generator
- [ ] Models support
  - [ ] Add azure open ai model support

<sub>From [SyncLinear.com](https://synclinear.com) | [R-274](https://linear.app/exploding-gradients/issue/R-274/improvements-in-v3-synthetic-data-generation)</sub>
Repository: explodinggradients/ragas
Issue: what does a context_relevancy of >1 mean?

I created a small Dataset of my own and observed that the context_relevancy>1
Can I please know what this means?

Hi @vibha0411 , this can be a bug. Can you please share the ragas version you're using and sample for which the context_relvancy score exceeded 1?

I am using ragas==0.0.11, following is the way i created the dataset:
```
from datasets import Dataset
def gen():
    yield {"question": "What is the Scope 1 Direct Emissions intensity for the year 2022?", "ground_truths":["The scope 1 emissions intensity for the year 2022 was 0.0157"],"answer":"The scope 1 emissions intensity for the year 2022 was 446,084","contexts":["Scope 2 Indirect Emissions in 2022 was 1,922,844","The scope 1 emissions for the year 2022 was 446,084","The scope 1 emmissions intensity was  0.0157"]}
ds = Dataset.from_generator(gen)
```
Result:
{'ragas_score': 0.0000, 'context_ relevancy': 1.9445, 'faithfulness': 0.0000, 'answer_relevancy': 0.9690, 'context_recall': 1.0000, 'harmfulness': 0.0000}

same issue here is this resolved or not ?


Hi @vibha0411 I was unable to reproduce your results 
`{'ragas_score': 0.0000, 'answer_relevancy': 0.9675, 'context_ relevancy': 0.9197, 'faithfulness': 0.0000, 'context_recall': 1.0000}`
This usually happens due stochastic nature of LLMs but I now have raised a fix for this. #105 

@Copticoder 
Once merged please reinstall ragas from the source
`pip install git+https://github.com/explodinggradients/ragas.git`

great ! 
can you please provide more details in the docs on these metrics ? like the methodology behind context_relevancy in details 

@Copticoder Sure, we are working on it. Will be able to update it before the next release.
Repository: explodinggradients/ragas
Issue: Support for multi-modal RAG evaluation (including images, tables, etc.)
Root cause: Documentation Deficiency

**Describe the Feature**
Support for multi-modal RAG evaluation (including images, tables, etc.)
Multi-modal RAG is becoming more important.



Closing as Multi-modal RAG evaluation for image+text contexts is implemented already. Also, has been migrated to the latest metrics collections.
Repository: explodinggradients/ragas
Issue: AttributeError: 'TestsetGenerator' object has no attribute 'generate_with_llama_index_docs'
Root cause: Documentation Deficiency

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When I try to run the code in "Compare LLMs using Ragas Evaluations" to create synthetic test data, I met following bug: AttributeError: 'TestsetGenerator' object has no attribute 'generate_with_llama_index_docs'

AttributeError                            Traceback (most recent call last)
Cell In[5], line 32
     25 distributions = {
     26     simple: 0.5,
     27     multi_context: 0.4,
     28     reasoning: 0.1
     29 }
     31 # generate testset
---> 32 testset = generator.generate_with_llama_index_docs(documents, 100,  distributions)
     33 testset.to_pandas()

AttributeError: 'TestsetGenerator' object has no attribute 'generate_with_llama_index_docs'

How can i check the attributes of TestsetGenerator and how to fix this?
Thank you so much for your attention for this issue! 
![Uploading 屏幕截图 2024-06-18 100837.png…]()



<!-- Greeting -->
@Zzrjin Hello there! I'm a bot that can help you with bugs, questions, and becoming a contributor. I'm here to assist you while you wait for a human maintainer.

<!-- Answer -->
The method name you're using is incorrect. The correct method name is `generate_with_llamaindex_docs`, not `generate_with_llama_index_docs`.

Here's the corrected code snippet:

```python
# generate testset
testset = generator.generate_with_llamaindex_docs(documents, 100, distributions)
testset.to_pandas()
```

To check the attributes of `TestsetGenerator`, you can inspect the class definition. The attributes of `TestsetGenerator` are:
- `generator_llm`
- `critic_llm`
- `embeddings`
- `docstore`

These attributes are initialized in the class constructor and used throughout the class methods <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/generator.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Problem solved. Thank you. 
But the code in the file "Compare LLMs using Ragas Evaluations" is still `generate_with_llama_index_docs_` , which may cause some confusion?
Repository: explodinggradients/ragas
Issue: [R-284] Use GPT-4o instead of GPT-4?
Root cause: Unimplemented Feature Gap

Would it be possible to use the GPT-4o model instead of GPT-4, would be much more cost effective and speedy for people.

Great product btw, thank you!

<sub>[R-284](https://linear.app/exploding-gradients/issue/R-284/use-gpt-4o-instead-of-gpt-4)</sub>

@arainey2022 thanks for the suggestion and the kind words 🙂 . 

We should do this this week @shahules786 ?

Hey @arainey2022 this was merged #1166