zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: text2sql examples runtime error [predicted_result: Input should be a valid string]
Root cause: Validation Gap

[X ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Error when calling `text2sql_experiment.arun`
```
Warning: Task failed with error: 
❌ Type validation errors for execution_accuracy:

   - predicted_result: Input should be a valid string (got:    Balance
0      0.0)
```

Ragas version: 0.4.1
Python version: 3.12

**Code to Reproduce**
```python
import asyncio
import os

import ragas_examples
from ragas_examples.text2sql.evals import text2sql_experiment, load_dataset

# Set your OpenAI API key
os.environ["OPENAI_API_KEY"] = "sk-3xxx"
os.environ["MODEL_NAME"] = "model-xx"

ragas_examples.text2sql.evals .__file__ = os.path.realpath(__file__)


async def run_evaluation():
    """Run text-to-SQL evaluation with direct code approach."""
    # Load dataset
    dataset = load_dataset()
    print(f"Dataset loaded with {len(dataset)} samples")

    prompt_file_name = f"{os.environ["MODEL_NAME"]}-prompt-v1"

    # Run the experiment
    results = await text2sql_experiment.arun(
        dataset,
        name=prompt_file_name,
        model=os.environ["MODEL_NAME"],
        prompt_file=None,
    )

    # Report results
    print(f"✅ {prompt_file_name}: {len(results)} cases evaluated")

    # Calculate and display accuracy
    accuracy_rate = sum(1 for r in results if r["execution_accuracy"] == "correct") / max(1, len(results))
    print(f"{prompt_file_name} Execution Accuracy: {accuracy_rate:.2%}")


# Run the evaluation
asyncio.run(run_evaluation())
```

**Error trace**
No stack print.

**Expected behavior**
No errors.


**Additional context**

The third param of method [`execution_accuracy`](https://github.com/vibrantlabsai/ragas/blob/84c038d9d479bccd91c9a0afd21861431535a837/examples/ragas_examples/text2sql/evals.py#L32) has no type defined. But the [discrete_metric decorator](https://github.com/vibrantlabsai/ragas/blob/84c038d9d479bccd91c9a0afd21861431535a837/src/ragas/metrics/decorator.py#L173) has a customer validation rule where validate param without specified type as a `str` one, this is the source code that throw `Validation error`
<br/><img width="1078" height="907" alt="Image" src="https://github.com/user-attachments/assets/10c0c896-79a8-4aaf-97c4-56f53fe6d31b" />
Repository: explodinggradients/ragas
Issue: [AG-UI] Error in experiments.py and ag_ui.ipynb
Root cause: Interface Contract Mismatch

[X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
[AG-UI] Running experiments.py or using latest "howto" .ipynb results in "Warning: Task failed with error: Cannot use aembed_text() with a synchronous client. Use embed_text() instead."

Ragas version: (unreleased)
Python version: 3.12

**Code to Reproduce**
Just run either the AG-UI experiments.py or the ag_ui.ipynb

**Error trace**
"Warning: Task failed with error: Cannot use aembed_text() with a synchronous client. Use embed_text() instead."

**Expected behavior**
I expect the experiments and notebook to run without errors. 


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->
Repository: explodinggradients/ragas
Issue: [Feature Request] Support for generating testsets from pre-chunked documents
Root cause: Unimplemented Feature Gap

Problem Description
I've noticed that when generating a testset using TestsetGenerator, Ragas currently applies its own chunking logic (e.g., HeadlineSplitter) to the input documents. This means that even if a user provides documents that are already pre-chunked according to their specific strategy, Ragas will re-chunk them.

In many real-world RAG applications, the chunking strategy is a critical part of the pipeline (e.g., semantic chunking, recursive character splitting with specific overlaps). To strictly evaluate the performance of these pipelines, it would be beneficial to generate testsets based on the exact chunks that are used in the application, without further modification by Ragas.

Proposed Solution
The rough idea is to introduce a way to pass these pre-chunked documents directly into the generator as chunks.

We could perhaps add a method like generate_with_chunks() or a flag that tells the generator "these are already chunks, please don't split them again". Internally, this would just treat the input documents as NodeType.CHUNK right away and skip the default splitting step.

Contribution
I'm interested in contributing this feature. I've explored the codebase and believe I can implement this functionality.

Would this be a welcome addition to Ragas? Please let me know your thoughts!

> I'm interested in contributing this feature. I've explored the codebase and believe I can implement this functionality.
> 
> Would this be a welcome addition to Ragas? Please let me know your thoughts!

Absolutely. Please go ahead.

Great, thanks! Could you assign this to me? I'll put up a draft PR soon.
Repository: explodinggradients/ragas
Issue: Support Mermaid diagrams in PDF generation
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Currently, the PDF export (`make build-docs-pdf`) doesn't render Mermaid diagrams because WeasyPrint is a static generator and doesn't run the client-side JavaScript required to render them. They just appear as raw code blocks in the final PDF.

I'd like to implement a fix for this, but I need input on the preferred approach regarding dependencies.

**Why is the feature important for you?**
Our documentation uses diagrams to explain architecture, and these are currently broken in the offline PDF version.

**Proposed Approaches**

**Option A: Kroki Plugin (Recommended for simplicity)**
Use `mkdocs-kroki-plugin` to render diagrams.
*   **Pros:** Very lightweight. No extra system dependencies. Keeps the build "Python-only".
*   **Cons:** Defaults to sending diagram code to the public `kroki.io` API.

**Option B: Offline Rendering (e.g., `mkdocs-mermaid-to-image`)**
Use a plugin that renders locally using a headless browser.
*   **Pros:** Fully offline, no data leaves the build environment.
*   **Cons:** Significantly heavier setup. Requires installing Node.js, Puppeteer, and Chrome/Chromium in the CI environment and for any dev building PDFs locally.


**Additional context**
I'm leaning towards Option A to keep the build simple (avoiding Node/Chrome requirements), but I wasn't sure if you had strict requirements about offline builds or external APIs.

Let me know which one you prefer, or if there's another approach I should look into!

Personally would prefer offline rendering. 

Do you want to work on this one as well?


Sure, I can  work on this. I will open a PR once I have it working.
Repository: explodinggradients/ragas
Issue: reformat:  Add pydantic for better error handling and debugging
Root cause: Unimplemented Feature Gap

Pydantic is a library that provides data validation and settings management using type annotations.

[Official documentation:](https://docs.pydantic.dev/latest/)

#Why use Pydantic?

- Powered by type hints — with Pydantic, schema validation and serialization are controlled by type annotations; less to learn, less code to write, and integration with your IDE and static analysis tools. 
- Speed — Pydantic's core validation logic is written in Rust. As a result, Pydantic is among the fastest data validation libraries for Python. 
- JSON Schema — Pydantic models can emit JSON Schema, allowing for easy integration with other tools. Learn more…
- Strict and Lax mode — Pydantic can run in either strict=True mode (where data is not converted) or strict=False mode where Pydantic tries to coerce data to the correct type where appropriate.
- Dataclasses, TypedDicts and more — Pydantic supports validation of many standard library types including dataclass and TypedDict
- Customisation — Pydantic allows custom validators and serializers to alter how data is processed in many powerful ways.
- Ecosystem — around 8,000 packages on PyPI use Pydantic, including massively popular libraries like FastAPI, huggingface, Django Ninja, SQLModel, & LangChain. 
- Battle tested — Pydantic is downloaded over 70M times/month and is used by all FAANG companies and 20 of the 25 largest companies on NASDAQ. If you're trying to do something with Pydantic, someone else has probably already done it.

Pydantic can provide better error handling and more visibility of classes, methods, and dependencies across the project.



would you give a few examples of how pydantic would help with error handling and debugging with respect to ragas.

if it is for LLM output parsing I agree. We are working out json for this since models are more likely to understand json errors than pydantic
Repository: explodinggradients/ragas
Issue: DEFAULT_TOKENIZER breaks code
Root cause: Architectural Constraint

[X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
In ragas/tokenizers line 125 breaks the code when in an env without internet

Ragas version: 0.4.2

**Code to Reproduce**
```
async def main() -> None:
    """Run the AnswerCorrectness metric on `sample`."""
    # The method `single_turn_ascore` returns a `float` (the score)
    # score = await scorer.single_turn_ascore(sample=sample, callbacks=None)
    # print(f"AnswerCorrectness score: {result:.4f}")
    metric = ContextPrecision(llm=llm)
    result = await metric.ascore(
        user_input="What is AI?",
        reference="AI is artificial intelligence...",
        retrieved_contexts=["Context 1", "Context 2"]
    )
    print(result)
```

When working in an environment without internet, but with a locally hosted LLM, this line breaks the code as it tries to connect to openAI. It can be fixed by using:

DEFAULT_TOKENIZER = None
**Error trace**

**Expected behavior**
A working module even without internet connection.


Hey @floorkouwenberg , I was able to reproduce this!

The issue is that ```DEFAULT_TOKENIZER``` on line 125 gets created at import time. When that happens, tiktoken tries to download the encoding if it's not already cached locally , and that fails in offline environments.

@anistark  I think lazy initialization could fix this , instead of creating the tokenizer at import time, we could defer it until it's actually used. Something like a  ```get_default_tokenizer()``` function. The only usage I found already wraps it in a ```default_factory```  lambda, so it should be a pretty safe change.

Would that approach work?

> The issue is that `DEFAULT_TOKENIZER` on line 125 gets created at import time. When that happens, tiktoken tries to download the encoding if it's not already cached locally , and that fails in offline environments.
> 
> [@anistark](https://github.com/anistark) I think lazy initialization could fix this , instead of creating the tokenizer at import time, we could defer it until it's actually used. Something like a `get_default_tokenizer()` function. The only usage I found already wraps it in a `default_factory` lambda, so it should be a pretty safe change.
> 
> Would that approach work?

Sounds decent approach. Do you wanna take it up?

Sure, I will go for it.

@floorkouwenberg Would you like to test if the PR #2545 fixes it for you?
Repository: explodinggradients/ragas
Issue: Can the experiment where ragas change from 0.3 to 0.4 achieve the previous "evaluate()" function?
Root cause: Unimplemented Feature Gap

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I noticed that after changing from version 0.3 to 0.4, the previous "evaluate()" function seems to no longer be supported. This function was capable of conducting batch-based indicator evaluations and automatically organizing the results, which was extremely convenient for beginners. Is it possible to achieve such a function now?

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us?
Repository: explodinggradients/ragas
Issue: faithfulness may be not faithful,because of the model's hallucinations.
Root cause: Validation Gap

Faithfulness, as an evaluation metric, may become less accurate when using the model's generation function, as it can be influenced by the model's hallucinations.
In the process of conducting extensive data evaluations, I encountered some anomalies in the assessment of AnswerCorrectness. Specifically, in cases where both the ground_truths and the answer had an AnswerSimilarity of 1, the AnswerCorrectness score was 0.6. However, when evaluated independently, the AnswerCorrectness score returned to 1. I suspect that the discrepancies in AnswerCorrectness are due to potential inaccuracies in the Faithfulness it relies on, and these inaccuracies may be attributed to the model's hallucinations.

Hi @PurpleLikeNero , thanks for rising the issue. Yes, this can happen and since all ragas metrics are based on LLMs, the quality if output is dependent on LLMs. Could you please share some more details like the model you're using and sample data if possible? 

sure,the LLMs i used named Qwen-7B,which is published by Alibaba Cloud. 

I'll let @shahules786 answer this but because of the model size/capabilities that would be expected

Hi, is there a workaround for this?
Or would you be knowing any other methods (even other than ragas) which could handle hallucinations?

I'm looking at options to identify hallucinations for my RAG system. I'm a bit sceptical to the idea of an LLM finding issues with an already LLM generated output. 
What would be the other options to identify hallucinations?

I looked at some tools like Vectara HHEM, but the context size is very less (512 tokens).
It would be really appreciated if someone could point me to a good direction.

Thanks!
Repository: explodinggradients/ragas
Issue: Importing LangchainLLM from ragas.llms throws validation error
Root cause: Validation Gap

This is what I was running

```
from langchain.chat_models import AzureChatOpenAI
from langchain.embeddings import OpenAIEmbeddings
from ragas.llms import LangchainLLM
```

Python==3.10.12
ragas==0.0.12

<img width="1316" alt="Screenshot 2023-11-08 at 4 38 47 PM" src="https://github.com/explodinggradients/ragas/assets/97539993/850fbe57-dcc0-4ba1-aa0a-1ec8f428ee7b">


Hi @Sckarge. Please add OPENAI_KEY to your environment variables and try again. 
```
import os
os.environ["OPENAI_API_KEY"] = "your-openai-key"
```
[refer here](https://docs.ragas.io/en/latest/getstarted/evaluation.html#get-started-evaluation)

this should be fixed with the latest release. feel free to reopen if this is still an issue :)
Repository: explodinggradients/ragas
Issue: Set versions of each dependency - new versions of llama-index breaks ragas
Root cause: External Dependency Breakage

**Describe the bug**
Installing ragas and installing the newest version of llama index breaks ragas - please add the correct versions of each dependency in the requirements

Ragas version: 0.0.18
Python version: 3.8

This is a problem with older versions of langchain, too.

this was fixed but I'll confirm this again before closing this
Repository: explodinggradients/ragas
Issue: Synthetic test data generation, but result was enpty
Root cause: Validation Gap

I' m tring to do "Synthetic test data generation", but But the result was empty. Can you help me solve this problem?

**Code:**
from langchain.document_loaders import PyPDFLoader
loader = PyPDFLoader("/home/data/_pdf/test_1.pdf")
documents_0 = loader.load()

from ragas.testset import TestsetGenerator

testsetgenerator = TestsetGenerator.from_default()
test_size = 12
testset = testsetgenerator.generate(documents_0, test_size=test_size)

**result:**
[/Users/opt/anaconda3/envs/py311_langchain/lib/python3.11/site-packages/ragas/testset/testset_generator.py:316](https://file+.vscode-resource.vscode-cdn.net/Users/wukaiyan/opt/anaconda3/envs/py311_langchain/lib/python3.11/site-packages/ragas/testset/testset_generator.py:316): UserWarning: Most documents are too short
  warnings.warn("Most documents are too short")
  0%|          | 0/2 [00:02<?, ?it/s]

TestDataset(test_data=[])



Hi @Gaia-Wu , can you share the ragas version?

@shahules786 

Name: ragas
Version: 0.0.19

From the warning here 
`serWarning: Most documents are too short
warnings.warn("Most documents are too short")`
most documents seem to be too short for going through QA evolution process. This might have caused. I would suggest to pass larger chunks and/or modify test distribution to `{'simple':1.0}` 

> From the warning here `serWarning: Most documents are too short warnings.warn("Most documents are too short")` most documents seem to be too short for going through QA evolution process. This might have caused. I would suggest to pass larger chunks and/or modify test distribution to `{'simple':1.0}`

@shahules786  Thanks for your guidance. I have solved this problem.
Repository: explodinggradients/ragas
Issue: Context (Average) Precision and LongContextReorder
Root cause: Documentation Deficiency

Hi, I noticed something interesting that I thought might be worth mentioning. 

There's been some attention around the paper "[Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172)"

LangChain has introduced a [LongContextReorder](https://python.langchain.com/docs/modules/data_connection/document_transformers/post_retrieval/long_context_reorder) class, which basically reorders the retrieved documents so that the least relevant documents are in the middle instead of at the end of the document list. The claim is that having the most relevant documents at the beginning and end is actually better for the LLM than to rank them in true ascending order.

The side effect of course is that `LongContextReorder` should have the unfortunate effect of lowering the Average Precision calculation.

I just thought I'd make a note of that in case anyone else came across this. 

Hey @austinmw , thank you for sharing this. I had similar thoughts after going through the paper. This is one of the reasons why I added the new [context_precicion](https://docs.ragas.io/en/latest/concepts/metrics/context_precision.html) metric which is rank-sensitive precision, Would love to know your thoughts on it. Here's a [reference](https://amitness.com/2020/08/information-retrieval-evaluation/)

Regarding finding in "lost in the middle" paper, I believe this behavior of the particular LLM is due the nature of finetuning data ( which makes LLM attention scores ignore tokens in the middle).  In my research & understanding it is always best to have relevant chunks at the top. 

I appreciate your genuine interest in our project, I would love to have a chat with you on this and possible contribution opportunities. Here's my [calendly](https://calendly.com/shahules/30min). 

 

Scheduled, looking forward to talking!
Repository: explodinggradients/ragas
Issue: Evaluation fails with openai version 1.2.3
Root cause: External Dependency Breakage

Evaluating metrics throws the following error with openai==1.2.3

```
File ".../python3.10/site-packages/langchain/chat_models/openai.py", line 73, in _create_retry_decorator
    openai.error.Timeout,
AttributeError: module 'openai' has no attribute 'error'
```

Fixed by rolling back to openai==0.28.1

Relevant langchain issue:
https://github.com/langchain-ai/langchain/issues/12967

Ragas version: 0.0.19
Python version: 3




I'm also facing the same issue. Any fix to this?

should be fixed with the latest release. will make a final check before closing this
Repository: explodinggradients/ragas
Issue: Latest dependency versions for getting open source LLMs to work
Root cause: External Dependency Breakage

Can someone please list which versions of openai, llama index, etc are currently required to get the documentation examples to work properly with open source LLMs? I've tried ragas evaluation and testsetgenerator with TGI and vLLM endpoints and run into missing dependencies. Having a tough time nailing down which ones currently work. The latest `llama-index` dependency throws `ModuleNotFoundError: No module named llama_index.node_parser.simple` when trying to run TestSetGenerator. I've rolled openai back to `openai==0.28.1` but this throws `ImportError: cannot import name 'AsyncAzureOpenAI' from 'openai`. Thanks!

Hi jenghub,

This is an issue with the latest version of llama_index. You can use llama-index==0.8.69.post1 to resolve this issue. Note: if you installing ragas first it will install llama_index with latest version. 
pip install llama-index==0.8.69.post1
pip install ragas
This order helped me resolve the issue.


@tinomaxthayil thank you, this seems to work for now! Although hitting other errors with evaluation and testsetgenerator which has been raised in other issues. Cheers!
Repository: explodinggradients/ragas
Issue: Answer relevancy always returns 1
Root cause: Algorithmic Error

**Describe the bug**
The Answer Relevancy Score calculation is malfunctioning when used with embedding models such as OpenAI or Vertex Gecko. Regardless of the actual similarity between answers, the score is consistently outputting a value of '1'. This issue has been observed since the latest software release and was not present in the earlier version (ragas==0.0.19).
Python version:
pyhton3.9




@ferdinandl007 : I also noticed this:

```python

fake_result = result.copy()
fake_result["result"] = "we are the champions"
print("fake result = ", fake_result["result"])
eval_result = faithfulness_chain(fake_result)
eval_result["faithfulness_score"]
```


which logs:

```
fake result =  we are the champions
1.0
```

I tried this:

```python
eval_questions = [
    "Can I work remotely?",
]

eval_answers = [
    "You must eat home-made pasta!",
]

examples = [
    {"query": q, "ground_truths": [eval_answers[i]]}
    for i, q in enumerate(eval_questions)
]

eval_result = faithfulness_chain(result)
eval_result["faithfulness_score"]
```


This logs `1.0`.. I don't understand.. am I missing something ?

Hey @younes-io , Please pass a context along with your question if you are trying to measure faithfulness. Checkout our quickstart. 

Hey @ferdinandl007 , Answer_relevancy needs some rework, I'll be on it this weekend :) 

@shahules786 I already did, I will open a separate issue if required.. I realized I was "polluting" this issue x) sorry @ferdinandl007 

Hey @younes-io , I would be happy to look at if you can open a PR :) We certainly can benefit from your help, Thank you.
Repository: explodinggradients/ragas
Issue: Broken link in the documentation

**Describe the bug**
The link referred to in [this section](https://docs.ragas.io/en/latest/howtos/integrations/langchain.html#introducing-ragasevaluatorchain) is broken:

RagasEvaluatorChain creates a wrapper around the metrics ragas provides (documented **[here]**(https://github.com/explodinggradients/ragas/blob/main/docs/metrics.md)), making it easier to run these evaluation with langchain and langsmith.

**Expected behavior**
Just update the link with the correct URL

Hey @younes-io , Nice catch. Would you like to contribute? 

@shahules786 Sure