zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Faithfulness and Response Relevancy have high parse error rate with non-OpenAI model
Root cause: Validation Gap

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
On the same dataset:
OpenAI has a 4% error rate in parsing faithfulness metric
Anthropic has ~40% error rate in parsing faithfulness metric
Llama3.1-70B-instruct (via TogetherAI) has a ~40% error rate in parsing faithfulness metric and ~20% error rate in parsing response relevance.

Ragas version: 0.22
Python version: 3.11

**Code to Reproduce**
Share code to reproduce the issue

**Error trace**

**Expected behavior**
Model-agnostic prompts / parsers provide equivalent (low) error rates


**Additional context**
I'll try to provide some more experimental results for context when I can

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Hey, @ahgraber Have you tried tuning/modifying default prompts using the `set_prompts` and `get_prompts` methods in ragas?

I haven't tried tuning the default prompt yet; trying to understand why I'm getting the errors:

It seems like the first step in `faithfulness` that decomposes into simple sentences (typically) works fine.
The problem occurs that when there are many decomposed sentences (20+), the models have a hard time responding with the proper schema for the nli verdicts - it seems like the number of statements overwhelms the schema instructions.  Unfortunately, I've run out of time at this point to continue investigating; my recommendation to my team will be to use OpenAI models as that seems to be the service this was primarily designed for.

Secondly (and this should probably be its own issue), when the "repair" prompt triggers, it escapes all json characters, and then re-escapes them to deeper and deeper depths every time the repair prompt is sent while also repeating the prompt and few-shot examples.
It's possible I don't notice the repair prompt working if it works the first time, but I see a lot of logs that look like [failed_repair.txt](https://github.com/user-attachments/files/17648480/failed_repair.txt) and ultimately fail out

would you be able to use something like https://github.com/Arize-ai/phoenix for viewing the traces? I wanted to check it there was maybe a out-of-context errors are happening?

but we can get on a call sometime to debug more too


I am running out of context sometimes, but that is not the root cause of the initial error loop.  I'll see if I can get to adding traces, but I'm not sure what timing looks like (might take a week or two)
Repository: explodinggradients/ragas
Issue: Update docs
GitHub labels: documentation

Rewrite docs 

- [ ]  reference-free evaluation
- [ ] End-to-end evaluation  
- [ ]  Detailed description of each metric, algorithm details, and example of high and low-scoring sample
- [ ] Test set generation 






You should probably update readme as your very first example throws an error:

`ValueError: The metric [context_recall] that that is used requires the following additional columns ['ground_truths'] to be present in the dataset.`

because you do not specify `ground_truths` in your example:

```
# prepare your huggingface dataset in the format
# Dataset({
#     features: ['question', 'contexts', 'answer'],
#     num_rows: 25
# })
```



@shahules786 this is finished right?
Repository: explodinggradients/ragas
Issue: Seed questions for guiding test generation
Root cause: Unimplemented Feature Gap

Is there a way to guide the RAGAS test generation process with specific seed questions as input, so that generated questions will be of similar nature to the seed questions? 


Hey, we have this in our roadmap. We will be pushing something along these lines in the coming months. I would love to understand your requirements more to make sure that we consider it too while building for it. Open to [Chat](https://cal.com/shahul-ragas/30min)?

Currently, the test set generation generates generalized results. For example, if it is give a menu as input, it generates general questions about the type of food in the menu (i.e. how is it prepared? what are the ingredients? what is the history?) Instead, I would like it to generate based on a set of guiding questions (such as what is the most expensive meal on the menu? What are the vegan options?)

In previous versions, it was possible to override simple.seed_question_prompt, but I see the code has changed quite a bit since. Is there a way to currently override the internal seed prompt?
Repository: explodinggradients/ragas
Issue: When transforms is a list type, there is no generate_execution_plan function
Root cause: Documentation Deficiency


When I try to use my local llm to generate testset, there are some problems.

**Describe the bug**
In this file: ragas/src/ragas/testset/transforms/engine.py,
on line 104, the codes are:
![图片](https://github.com/user-attachments/assets/26e0b4ee-2182-46a0-94c2-21b07eb6ef2e)
but there is no generate_execution_plan function.(The existing generate_execution_plan function is for Parallel.)

Ragas version: 0.2.4
Python version: 3.8.19


@frr163 I'm not sure I understood the question but every transforms has an execution plan https://github.com/explodinggradients/ragas/blob/0415a2dd313f9991f9d41f8d78f55346a4a09235/src/ragas/testset/transforms/base.py#L76-L93

Prallel has a special one where it combines all the plans from the call and executes it together

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Can I control the concurrency of calling LLM
Root cause: Unimplemented Feature Gap

My LLM was based  on L20 devices, when generate Test Data It was unexpectedly terminated[ Aborted request]，Is it due to too much concurrency？


![image](https://github.com/user-attachments/assets/c4bb1e8f-918d-4b6d-8a65-31db17a2279e)
Repository: explodinggradients/ragas
Issue: ModuleNotFoundError: No module named 'llama_index.core'
Root cause: Validation Gap

[x ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Using version 0.2.4 requires llama index as dependency

Ragas version: 0.2.4
Python version: 3.10.13

**Code to Reproduce**
Any sample code

**Error trace**
Traceback (most recent call last):
  File "/Users/markus/Code/xyz/evaluation/ragas_evaluation.py", line 4, in <module>
    from ragas.metrics import (
  File "/Users/markus/Code/xyz/.venv/lib/python3.10/site-packages/ragas/__init__.py", line 2, in <module>
    from ragas.evaluation import evaluate
  File "/Users/markus/Code/xyz/.venv/lib/python3.10/site-packages/ragas/evaluation.py", line 11, in <module>
    from llama_index.core.base.llms.base import BaseLLM as LlamaIndexLLM
ModuleNotFoundError: No module named 'llama_index.core'


**Expected behavior**
Llama index only as optional dependency

**Additional context**
Problem introduced in 0.2.4, 0.2.3 works without issues.


Yes! Same problem - you can see in version 0.2.4 it forces you to install LLamaIndex before using evaluation.py

Version 0.2.4
https://github.com/explodinggradients/ragas/blob/v0.2.4/src/ragas/evaluation.py#L11

Version 0.2.3
https://github.com/explodinggradients/ragas/blob/v0.2.3/src/ragas/evaluation.py

hey @sam-h-long @cirezd  that was a bug on our end - this has been removed with v0.2.5

I'm closing this now but let me know if this is still a problem
Repository: explodinggradients/ragas
Issue: testset_generation：the max_wait not works
Root cause: Configuration Error

  runconfig = RunConfig(
        timeout=120, max_retries=1, max_wait=120, max_workers=1, log_tenacity=True
    )
    testset = generator.generate(
        testset_size=args.test_size,
        query_distribution=distributions,
        run_config=runconfig,
        with_debugging_logs=True,
        raise_exceptions=True,
    )

The max_workers seems not work! Frequent calls to LLM caused it to freeze


![image](https://github.com/user-attachments/assets/6cff3300-4931-4263-a57a-c764f69334c1)


hey @jackqdldd which LLM, LLM provider are you using?
Repository: explodinggradients/ragas
Issue: Timeout Error
Root cause: Configuration Error

ERROR:ragas.executor:Exception raised in Job[2]: TimeoutError()
ERROR:ragas.executor:Exception raised in Job[7]: TimeoutError()
ERROR:ragas.executor:Exception raised in Job[10]: TimeoutError()
ERROR:ragas.executor:Exception raised in Job[15]: TimeoutError()
....

I use GPT-4o and encountered many timeout error, any ideas?


I have reduced the number of timeout error by adjusting the max_workers to 60 and time_out parameter to 120 mentioned in [https://docs.ragas.io/en/stable/howtos/customizations/_run_config/?h=time](url)

But still some error existing, I think 120 is a very long time. Any ideas?

> max_workers

Further reducing max_workers solved the same issue for me. I set it to be 8 for my locally hosted models.

hey anothe approach is `batch_size` argument for `evalaute()` https://docs.ragas.io/en/stable/references/evaluate/#ragas.evaluation.evaluate

something else you could try? easier to reason about IMO

reducing max_workers to 10 and increasing the timeout to 120 seconds also worked for me. Even with a 120-second timeout, I could still get a parallelism of 8-9 out of 10, which might indicate that just a few workers were taking a long time to process.

Hi, has anyone resolved the Ragas timeout errors? I've trying many configuration settings and still get them.
Repository: explodinggradients/ragas
Issue: Bug in Testsetgeneration.ipynb
Root cause: Documentation Deficiency

Hi Devs, 

I found a bug in the notebook file. https://github.com/explodinggradients/ragas/blob/main/docs/Testsetgeneration.ipynb
When running this code: `from ragas.testset import TestsetGenerator`
there is an error because the `os.environ["OPENAI_API_KEY"] ="sk-xxx"` is set afterwards. 
It should be before we do import, and then it works. 

not sure where the fix should be done, in the TestsetGenerator, or in the notebook. 
Im a beginner in this. 
Thank you!

Nice catch @stepkurniawan 
Resolved #165
Repository: explodinggradients/ragas
Issue: Duplicate Questions and Spelling Mistakes in Generated Testset
Root cause: Unimplemented Feature Gap

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
When using the TestsetGenerator from the ragas.testset module, I am encountering the following issues:

Duplicate questions: The generated test set often contains repeated questions.
Spelling mistakes: The generated questions contain spelling errors (e.g., "Presidnet" instead of "President", "spesial meting" instead of "special meeting").

**Code Examples**

# Initialize the LLM wrapper
generator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4"))

# Initialize the Embeddings wrapper
generator_embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

from ragas.testset import TestsetGenerator

generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
dataset = generator.generate_with_langchain_docs(docs, testset_size=5)


**Additional context**

Sample Output:
What does the term 'The Site of the University of Missouri' refer to according to the Board Bylaws?
What is the role of the Presidnet in the University of Missouri as per the Board of Curators?
Who can call a spesial meting of the Board?
What are the responsibilities of the Board of Curators as per the Board Bylaws?
What are the responsibilities of the Board of Curators at the University of Missouri?
What are the responsibilities of the Board of Curators at the University of Missouri?



Hey @Jayashree-kalabhavi 
1) duplicate questions: we will take a look at this. This is related to one the issues on the roadmap
2) spelling mistakes: they are induced because of the query style = misspelled queries. As of now it's not fully configured by user. But we will add that too in the roadmap. 

Thanks for reporting. 

Spelling mistakes also in the answers. How to influence the prompt, where does it pull the instruction from to make spelling mistakes?

> Hey [@Jayashree-kalabhavi](https://github.com/Jayashree-kalabhavi)
> 
> 1. duplicate questions: we will take a look at this. This is related to one the issues on the roadmap
> 2. spelling mistakes: they are induced because of the query style = misspelled queries. As of now it's not fully configured by user. But we will add that too in the roadmap.
> 
> Thanks for reporting.

Hi, is the query_style configurable by the user now? Has this been added?
Repository: explodinggradients/ragas
Issue: AttributeError: object has no attribute 'sentences'
Root cause: Validation Gap

[X] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I constructed a simple RAG with LLM for the evaluation. Got some NaN and many 0.0 in the results. Also saw this error during the evaluation:

ERROR:ragas.executor:Exception raised in Job[0]: AttributeError('StringIO' object has no attribute 'sentences')

**Code Examples**
Here is the sample code:

```
ragas_data = [
    {
        "user_input": entry["user_input"],
        "response": entry["response"],
        "retrieved_contexts": entry["retrieved_contexts"],
        "reference": entry["reference"]
    }
    for entry in sample_dataset
]

evaluation_dataset = EvaluationDataset.from_list(ragas_data)

evaluator_llm = LangchainLLMWrapper(ChatCohere(cohere_api_key=COHERE_API_KEY, model="command-r-plus-08-2024"))
    results = evaluate(
        dataset = dataset,
        metrics=[
            Faithfulness(llm=evaluator_llm),
            LLMContextRecall(llm=evaluator_llm),
            LLMContextPrecisionWithReference(llm=evaluator_llm),
            FactualCorrectness(llm=evaluator_llm)
        ],
        llm = evaluator_llm
    )

    df = results.to_pandas()
```

The output of the df shows faithfulness as Nan, the llm_context_precision_with_reference with 0.0, while the context_recall is 0.5, and factual_correctness 0.89.

During the evaluation, it printed the error:
ERROR:ragas.executor:Exception raised in Job[0]: AttributeError('StringIO' object has no attribute 'sentences')

This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 


closed with https://github.com/explodinggradients/ragas/pull/1864 will be released with v0.2.12 🙂

I'm closing this for now but if the issue is still persisting, please do let me know - really sorry about the delay
Repository: explodinggradients/ragas
Issue: ToolCallAccuracy raises ZeroDivisionError when called without any arguments
Root cause: Validation Gap

**Describe the bug**
The function _get_arg_score raises a division by zero error when the refs dictionary is empty. This condition occurs when a tool is called without any arguments, which is a valid scenario, especially during a handoff operation.

**Code to Reproduce**
```python
from ragas.metrics import ToolCallAccuracy
from ragas.dataset_schema import MultiTurnSample
from ragas.messages import ToolCall, AIMessage
from langchain_openai import ChatOpenAI

ragas_trace = [AIMessage(content='', metadata=None, type='ai', tool_calls=[ToolCall(name='transfer_to_sales_agent', args={})])]

sample = MultiTurnSample(
    user_input=ragas_trace,
    reference_tool_calls=[
        ToolCall(name="transfer_to_sales_agent", args={})
    ],
)

tool_accuracy_scorer = ToolCallAccuracy()
tool_accuracy_scorer.llm = ChatOpenAI(model="gpt-4o-mini")
await tool_accuracy_scorer.multi_turn_ascore(sample)
```

**Error trace**
```shell
File ~/miniconda3/envs/agents/lib/python3.10/site-packages/ragas/metrics/_tool_call_accuracy.py:48, in ToolCallAccuracy._get_arg_score(self, preds, refs, callbacks)
     40     if arg in preds:
     41         score += await self.arg_comparison_metric.single_turn_ascore(
     42             SingleTurnSample(
     43                 response=str(preds[arg]), reference=str(refs[arg])
     44             ),
     45             callbacks,
     46         )
---> 48 return score / len(refs.keys())

ZeroDivisionError: float division by zero
```

**Expected behavior**
- If both refs and preds are empty, the function should return a score of 1.0.
- If refs is empty but preds is not, return 0.0.
Repository: explodinggradients/ragas
Issue: For some RAGAS metrics score is not accurate:
Root cause: Algorithmic Error

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
what is unclear to you? What would you like to know?

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 


from ragas.metrics import context_precision,answer_relevancy,faithfulness,context_recall,answer_similarity,answer_correctness,context_entity_recall
from ragas.metrics._aspect_critic import harmfulness

Faithfulness: I have a context which does not contain answer to the question and response given by llm is also doesn't contain information for the given question in the context why the score is 0 and for the same type of case it is 1 for the same question with different context (which also not contain answer for the question).

Answer correctness: ground truth and response from llm is not matching but the score is still 0.88 but it is 0.11 for the same for the same ground truth and response. why there is differences in the scores for same cases?

Harmfulness: the question and response does not contain words which cause harm but still the score is 1 and for the same question and response it is 0 again. why scores differ?

I find scores are inconsistent. is it so? why there is differences in score for the same cases
Repository: explodinggradients/ragas
Issue: Prompt Error, TimeOut error
Root cause: Validation Gap

2024-11-20 16:06:08,086 - ERROR - Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
2024-11-20 16:06:08,093 - ERROR - Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
2024-11-20 16:06:08,094 - ERROR - Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
2024-11-20 16:06:08,094 - ERROR - Prompt context_recall_classific[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
This two errors are continuously appearing when evaluating on HuggingFace models and HF Datasets

Ragas version:latest
Python version:3.12

**Code to Reproduce**


**Error trace**

**Expected behaviour**:
I expect no output parser errors!!


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Hello guys ; any update on this issue? I ma still facing it blocking my pipeline..:)

Hi @giambascientist86, I encountered the same issue and managed to resolve it by using a callback to capture the format of the prompt and the model's output. I actually came across a great solution in #1729, which worked really well.

Here's how I did it:

```python
from langchain_core.callbacks import BaseCallbackHandler
from ragas import evaluate,RunConfig

class TestCallback(BaseCallbackHandler):

    def on_llm_start(self, serialized, prompts, **kwargs):
        print(f"**********Prompts*********:\n {prompts[0]}\n\n")

    def on_llm_end(self, response, **kwargs):
        print(f"**********Response**********:\n {response}\n\n")

score = evaluate(dataset,metrics=[faithfulness],
                 llm=llm,
                 embeddings=azure_embeddings,
                 raise_exceptions=True,
                 callbacks=[TestCallback()],
                 run_config=RunConfig(timeout=10,max_retries=1,max_wait=60,max_workers=1)
                )
score.to_pandas()
```
Hope it helps!

Also I use ragas == 0.2.6
Repository: explodinggradients/ragas
Issue: TestsetGenerator() not producing the "response" field
Root cause: Documentation Deficiency


Hi! I am trying to generate a test set using the following code:

Ragas version: 0.2.3
Python version: 3.11.9

```
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.testset import TestsetGenerator

def generate_test_set(llm,
                      embedder,
                      documents
                    ):

    generator_embeddings =  LangchainEmbeddingsWrapper(embedder)
    
    generator_llm = LangchainLLMWrapper(llm)
    
    generator = TestsetGenerator(llm=generator_llm,
                                 embedding_model=generator_embeddings)
    
    test_dataset = generator.generate_with_langchain_docs(documents, testset_size=1)
    
    return test_dataset


test_dataset = generate_test_set(llm=llm, embedder=embedder, documents=documents)
```
My documents variable is a list of _langchain_core.documents.base.Document_. each of them containing a page_content and a metadata={'title'} keys.

However I only obtain the following output columns:
```
user_input | reference_contexts | reference | synthesizer_name
```
I suppose "reference_contexts" is the "retrieved_contexts" needed by the metrics later on (so I changed the column name), but I don't get the "response" field, needed for example for FactualCorrectness and SemanticSimilarity metrics calculation.

Is there a way to get "response" and not just "reference"?  Otherwise I will get a semantic similarity of 100% by duplicating "reference" as "response".

Best regards,
Rubén.




"response" is the generated from your model not the test generator, same for "retrieved_contexts". Your evaluation dataset should have both the fields from test generator `user_input | reference_contexts | reference` and added to them values returned from your model: `response | retrieved_contexts`

It is not a bug, even here in the documentation:https://docs.ragas.io/en/stable/howtos/integrations/_llamaindex/ 
You have clearly stated the output of the test_set generator. You can easily transform test_set to the expected format, but you need to remember about answers from your model which are necessary to have correct evaluation :)