zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Unable to create Test data with TestsetGenerator.
Root cause: Interface Contract Mismatch

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Unable to create Test data with TestsetGenerator.

Ragas version:0.3.5
Python version: 3.13.3

```sh
uv venv python 
Using CPython 3.13.3
Creating virtual environment at: python
Activate with: source python/bin/activate
```

**Code to Reproduce**

```
from langchain_community.document_loaders import DirectoryLoader
import os
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI
from ragas.embeddings import OpenAIEmbeddings
import openai
from ragas.testset import TestsetGenerator

path = "Sample_Docs_Markdown/"
loader = DirectoryLoader(path, glob="**/*.md")
docs = loader.load()

generator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
openai_client = openai.OpenAI()
generator_embeddings = OpenAIEmbeddings(client=openai_client)

generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
print(dataset.to_pandas())

```
**Error trace**

```
unable to apply transformation: Cannot use aembed_text() with a synchronous client. Use embed_text() instead.
Traceback (most recent call last):                                                          
  File "/home/telesoho/prjs/ragas/gen_testset/gen_testset.py", line 23, in <module>
    dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
  File "/home/telesoho/prjs/ragas/src/ragas/testset/synthesizers/generate.py", line 189, in generate_with_langchain_docs
    apply_transforms(kg, transforms)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/telesoho/prjs/ragas/src/ragas/testset/transforms/engine.py", line 108, in apply_transforms
    transform.generate_execution_plan(kg),
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^
  File "/home/telesoho/prjs/ragas/src/ragas/testset/transforms/engine.py", line 41, in generate_execution_plan
    coroutines.extend(transformation.generate_execution_plan(kg))
                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^
  File "/home/telesoho/prjs/ragas/src/ragas/testset/transforms/relationship_builders/cosine.py", line 93, in generate_execution_plan
    raise ValueError(f"Node {node.id} has no {self.property_name}")
ValueError: Node 90fa920a-49c0-43e2-97ac-ae76f33866b6 has no summary_embedding
```

**Expected behavior**

**Additional context**

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->
Repository: explodinggradients/ragas
Issue: make heavy  dependencies optional
Root cause: Architectural Constraint

currently we have a few heavy dependencies like sentence-transformer and torch, we can reduce this for smaller installables

This will require

- [ ] Removing strictness &[ sentence agreements](https://github.com/explodinggradients/ragas/blob/1b2c79b58c632bbb407a72b3f0a376503b8a17ae/src/ragas/metrics/context_precision.py#L49) from context relevancy
- [ ] Adding bi-encoder methods in [answer similarity  ](https://github.com/explodinggradients/ragas/blob/1b2c79b58c632bbb407a72b3f0a376503b8a17ae/src/ragas/metrics/answer_similarity.py#L45)

currently size of library is very big, removing these dependencies will make it more usable.

fixed now
Repository: explodinggradients/ragas
Issue: How to use Amazon Bedrock as LLM/Embeddings after LangChain Wrappers are removed
Root cause: Unimplemented Feature Gap

[ x ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
In pull request #2253, LangChainLLMWrapper and LangchainEmbeddingsWrapper have been deprecated, but could you tell me about alternative approaches when using Amazon Bedrock? It seems that instructor_llm_factory and embedding_factory don't support Bedrock yet. Will Bedrock support be added to these factories in the future?

**Code Examples**
None

**Additional context**
- This question is related to belows:
  - https://github.com/explodinggradients/ragas/pull/2253


thank you so much for raising this @licux  - this is something new we are exploring and we would highly appreciate your feedback on this 🙌🏽 .

so currently you can leverage `InstructorLLM` base class an manually create the instructor client as so

```py
# create the instructor client
import boto3
import instructor

bedrock_client = boto3.client('bedrock-runtime')
client = instructor.from_bedrock(bedrock_client)

# use that to create InstructorLLM from ragas
from ragas.llms.base import InstructorLLM

llm = InstructorLLM(
    model="anthropic.claude-3-sonnet-20240229-v1:0",
    provider="bedrock",
    client=client,
)

llm.generate("Extract: Jason is 25 years old name", response_model=User)

# use that to create metrics
from ragas.metrics import DiscreteMetric, NumericMetric

summary_accuracy = DiscreteMetric(
    name="summary_accuracy",
    prompt="Evaluate if given summary is accurate for the user's query: {user_input}? {response}",
    allowed_values=(0, 1),
)

summary_accuracy.score(**test_data, llm=llm)
# MetricResult(value=1, reason='The given summary accurately captures the key points from the text, including the 8% growth in Q3 2024, the strong performance in the Asian market as a driving factor, the role of strategic marketing and product localization in this success, and the positive outlook for continued growth in the next quarter.')
```

we should also add "bedrock" as one of the supported providers @anistark maybe? 

the main change we are trying to do is depend on the clients the provider gives directly instead of using langchain and since we need structured outputs we use Instructor to do this

Thank you!

As below LangChainLLMWrapper will be deprecated so Ragas users using AWS must migrate to this feature. We strongly hope this to support Amazon Bedrock LLMs & embeddings!
https://github.com/explodinggradients/ragas/pull/2253

hey @minorun365 we'll definitely be adding first class support for it too 🙂  

hey @jjmachan , any updates on adding native support for bedrock?
Repository: explodinggradients/ragas
Issue: RFC: Refactor metrics module - Create metrics.core for better separation of abstractions
GitHub labels: enhancement

RFC

module-metrics

## 📋 Proposal

I'm proposing to refactor the metrics module to create a clearer separation between core abstractions and specific metric implementations.

## 🎯 Motivation

Currently, the `src/ragas/metrics/` directory mixes core abstractions with specific implementations, making it harder to:
- Understand which files are foundational vs. implementations
- Navigate the codebase for new contributors
- Maintain clear architectural boundaries

## 📁 Current Structure

The metrics directory contains ~40 files with two categories:

**Core abstractions (files without '_' prefix):**
- `base.py` - Base metric classes and interfaces
- `result.py` - MetricResult class
- `decorator.py` - Metric decorator factory
- `validators.py` - Validation utilities
- `discrete.py` - DiscreteMetric class
- `llm_based.py` - LLM-based metric primitives
- `numeric.py` - NumericMetric class
- `ranking.py` - RankingMetric class
- `utils.py` - Utility functions
- `quoted_spans.py` - Quoted span utilities

**Specific implementations (files with '_' prefix):**
- 30+ files like `_answer_correctness.py`, `_faithfulness.py`, etc.

## 🚀 Proposed Solution

Create a `metrics.core` submodule:

```
src/ragas/metrics/
├── core/
│   ├── __init__.py
│   ├── base.py
│   ├── result.py
│   ├── decorator.py
│   ├── validators.py
│   ├── discrete.py
│   ├── llm_based.py
│   ├── numeric.py
│   ├── ranking.py
│   ├── utils.py
│   └── quoted_spans.py
├── _answer_correctness.py
├── _faithfulness.py
├── ... (other implementations)
└── __init__.py
```

## ✅ Benefits

1. **Clearer separation**: Explicit distinction between abstractions and implementations
2. **Better organization**: Core functionality grouped logically
3. **Improved discoverability**: Developers can easily find foundational components
4. **Reduced cognitive load**: Clear hierarchy between core and specific metrics

## ⚠️ Considerations

1. **Breaking changes**: Existing imports would need updating
2. **Backward compatibility**: We should maintain compatibility with:
   ```python
   # Old way (should still work)
   from ragas.metrics.base import Metric
   from ragas.metrics.result import MetricResult
   
   # New way
   from ragas.metrics.core import Metric, MetricResult
   ```

## 💭 Questions for Discussion

1. Is this separation valuable enough to justify the refactoring effort?
2. Should we use a different name than "core"? (alternatives: "base", "abstractions", "primitives")
3. How should we handle the migration? Phase it over multiple releases?
4. Are there other files that should be considered "core" that I've missed?

## 🔄 Migration Strategy (if approved)

1. Create the `metrics.core` module with all core files
2. Update internal imports throughout the codebase
3. Add backward compatibility imports in `metrics.__init__.py`
4. Deprecation warnings for old import paths (remove in future major version)
5. Update all documentation and examples

Looking forward to feedback from the maintainers and community!

cc: @jjmachan
Repository: explodinggradients/ragas
Issue: Evaluations are ~10 min for 838 calls despite 5k RPM and 1M TPM Limits — how to ensure async with llm_factory (gpt-4o-mini)?
Root cause: Documentation Deficiency

**Your Question**
<img width="866" height="34" alt="Image" src="https://github.com/user-attachments/assets/1dfacf4e-27a0-4ae9-b335-0c2c33029385" />

If I were to make these calls myself with a custom asynchronous client, it takes less than a minute. None of the evaluations depend on each other, so I do not see the problem of sending all the requests out immediately without waiting for the ones before to come back. Keep in mind that I do not currently have a metric that requires embeddings to be computed. 

- Does llm_factory(model_name) return an async-capable client by default?

- How can I guarantee the async codepath is used (e.g., pass an httpx.AsyncClient / pool) when using llm_factory?

- Are there internal concurrency caps (e.g., connection pool limits, batch/phase scheduling) that would keep wall time near 10 minutes even with RunConfig(max_workers=5000)?

- Any recommended settings (max workers, batch size, client pool sizes) for high-RPM runs?

 
> * Does llm_factory(model_name) return an async-capable client by default?

Yes.

> * How can I guarantee the async codepath is used (e.g., pass an httpx.AsyncClient / pool) when using llm_factory?

Should be default behaviour. Try the latest version.

> * Are there internal concurrency caps (e.g., connection pool limits, batch/phase scheduling) that would keep wall time near 10 minutes even with RunConfig(max_workers=5000)?

There's a concurrency limit at max_workers = 16

You can increase your API limits in config:

```py
run_config = RunConfig(
    max_workers=100,
    timeout=120,
)
```

> * Any recommended settings (max workers, batch size, client pool sizes) for high-RPM runs?

Try to play around the config:

```py
run_config = RunConfig(
    max_workers=100,
    timeout=120,
    max_retries=5,
    max_wait=30,
)
```


It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: [Proposal] We built a Ragas Eval provider for Llama Stack. We would like to contribute it to the Ragas community.

Hello Ragas community! 👋 Diego here from Red Hat's TrustyAI team.

We built a Ragas Eval provider for Llama Stack. Would this be of interest to the Ragas community?

**What is this?**

This provider allows you to run all of Ragas' powerful evaluation metrics via Llama Stack's standardized evaluation API. It comes in two flavors:
  - Inline: Runs evaluations in the same process as your Llama Stack server
  - Remote: Leverages Kubeflow Pipelines for distributed evaluation workloads

**Why did we do it?**

We are adopting Llama Stack as the opinionated organization of GenAI APIs for inference, guardrailing, evaluation, etc. This integration allows us to bring Ragas into it:
  - Unified Interface: Access Ragas evaluations alongside other Llama Stack components (inference, embeddings, datasets)
  - Scalability: The remote provider enables running Ragas evaluations in Kubernetes environments via Kubeflow Pipelines

**Current state**

The provider is functional and includes support for both local and remote / KFP execution. It wraps the inference API provided by Llama Stack so that generations and embeddings required by Ragas are routed to the Llama Stack server.

**Proposal / Discussion**

  We'd love for this provider to become part of Ragas - perhaps under an integrations repo or within a subfolder? That way it can grow alongside Ragas. Perhaps we can separate the Llama Stack piece from the Kubernetes piece so folks can also use the Kubeflow pipeline by itself and make Llama Stack optional?

Would love to hear your thoughts and thank you for Ragas!


  Current Repository: https://github.com/trustyai-explainability/llama-stack-provider-ragas
  PyPI: https://pypi.org/project/llama-stack-provider-ragas/
Repository: explodinggradients/ragas
Issue: Can't patch loop of type <class 'uvloop.Loop'>
Root cause: External Dependency Breakage

when i run evaluation, i got this problem. I try to set the env by "UVICORN_NO_UVLOOP=1", but it didn't work. How can I fix it
`evaluate(
            llm=self.ev_llm,
            batch_size=20,
            dataset=self.dataset,
            metrics=metrics,
            run_config=run_config
        )`


Can't patch loop of type <class 'uvloop.Loop'>

@snailmiss Thanks for bringing this up. It's a tricky situation.

Could you care to test if PR #2369 fixes it for you?
Repository: explodinggradients/ragas
Issue: add token parser to generate
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Missing props **(token_usage_parser)** in the method
- generate_with_langchain_docs
- generate_with_llamaindex_docs

**Why is the feature important for you?**
I can count how many tokens used for test generation

```diff
@@ -108,6 +108,7 @@ class TestsetGenerator:
         query_distribution: t.Optional[QueryDistribution] = None,
         run_config: t.Optional[RunConfig] = None,
         callbacks: t.Optional[Callbacks] = None,
+        token_usage_parser: t.Optional[TokenUsageParser] = None,
         with_debugging_logs=False,
         raise_exceptions: bool = True,
         return_executor: bool = False,
@@ -133,6 +134,9 @@ class TestsetGenerator:
             Configuration for the generation run, by default None
         callbacks : Optional[Callbacks], optional
             Callbacks to use during generation, by default None
+        token_usage_parser : Optional[TokenUsageParser], optional
+            Parse the LLMResult object and return a TokenUsage object. This is used to
+            calculate the cost of the generation process.
         with_debugging_logs : bool, optional
             Whether to include debug logs, by default False
         raise_exceptions : bool, optional
@@ -196,6 +200,7 @@ class TestsetGenerator:
             query_distribution=query_distribution,
             run_config=run_config,
             callbacks=callbacks,
+            token_usage_parser=token_usage_parser,
             with_debugging_logs=with_debugging_logs,
             raise_exceptions=raise_exceptions,
             return_executor=return_executor,
@@ -211,6 +216,7 @@ class TestsetGenerator:
         query_distribution: t.Optional[QueryDistribution] = None,
         run_config: t.Optional[RunConfig] = None,
         callbacks: t.Optional[Callbacks] = None,
+        token_usage_parser: t.Optional[TokenUsageParser] = None,

```
Repository: explodinggradients/ragas
Issue: Remove unused code in engine.py
GitHub labels: module-testsetgen



The file `src/ragas/testset/transforms/engine.py` contains unused code that should be removed to improve code maintainability and reduce dependencies.

## Unused Code Identified

### 1. Unused function: `run_coroutines()`
**Location:** Lines 48-64 in `src/ragas/testset/transforms/engine.py`

This function is defined but never called anywhere in the codebase. The functionality has been replaced by `run_async_tasks()` from `ragas.async_utils`.

```python
async def run_coroutines(
    coroutines: t.Sequence[t.Coroutine], desc: str, max_workers: int
):
    """
    Run a sequence of coroutines in parallel.
    """
    for future in tqdm(
        as_completed(coroutines, max_workers=max_workers),
        desc=desc,
        total=len(coroutines),
        # whether you want to keep the progress bar after completion
        leave=False,
    ):
        try:
            await future
        except Exception as e:
            logger.error(f"unable to apply transformation: {e}")
```

### 2. Unused imports
**Location:** Lines 6-8 in `src/ragas/testset/transforms/engine.py`

The following imports are only used by the unused `run_coroutines()` function:
- `from tqdm.auto import tqdm`



## Proposed Solution

Remove:
The unused imports:
   - Remove `from tqdm.auto import tqdm` (line 6)
---


**Files Affected:**
- `src/ragas/testset/transforms/engine.py`
Repository: explodinggradients/ragas
Issue: ValidationError: 2 validation errors for ThemesPersonasInput when generating test dataset
Root cause: Validation Gap

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When using `generator.generate_with_langchain_docs(docs, testset_size=10)` in Ragas, a `ValidationError: 2 validation errors for ThemesPersonasInput` is raised. The error occurs even when following the documentation examples for test generation on AWS



Ragas version:
0.3.7
Python version:
python 3.13

**Code to Reproduce**
Share code to reproduce the issue
```
from ragas.testset import TestsetGenerator
from langchain_aws import ChatBedrockConverse
from langchain_aws import BedrockEmbeddings
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper

generator_llm = LangchainLLMWrapper(ChatBedrockConverse(
    region_name=config["region_name"],
    base_url=f"https://bedrock-runtime.{config['region_name']}.amazonaws.com",
    model=config["llm"],
    temperature=config["temperature"],
))
generator_embeddings = LangchainEmbeddingsWrapper(BedrockEmbeddings(
    region_name=config["region_name"],
    model_id=config["embeddings"],
))
generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
```
**Error trace**

ValidationError: 2 validation errors for ThemesPersonasInput
themes.0
  Input should be a valid string [type=string_type, input_value=('Bottlerocket', 'Bottlerocket'), input_type=tuple]
    For further information visit https://errors.pydantic.dev/2.12/v/string_type
themes.1
  Input should be a valid string [type=string_type, input_value=('SSH', 'SSH'), input_type=tuple]
    For further information visit https://errors.pydantic.dev/2.12/v/string_type

**Expected behavior**
It should generate the dataset


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Thanks for reporting @clementrey-dev 
Was able to reproduce it. Will check on a fix.

@clementrey-dev Would you be able to check if PR #2377 works for you?
Repository: explodinggradients/ragas
Issue: Is it possible to set system prompt words?
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
During the evaluation process, we often need to set a specific role information and strict system prompts for the LLM. Is there any exposed interface that allows us to set the system prompt words before the evaluation?

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 


Hi @jietang00723-cloud  👋🏼 

Could you please share more in your code where you're stuck?

In general, you can provide custom prompts right now like so:

```py
metric = AnswerCorrectness()
metric.prompt = YourCustomPrompt()
```

Also, Metrics use `BasePrompt` classes from `src/ragas/prompt/` which define the system message, user template, and output format. You can create custom prompt classes by extending `BasePrompt`.

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Disable LangSmith
Root cause: Configuration Error

Hi, I'm getting the following errors logged when running eval:

```
Connection error caused failure to post http://localhost:1984/runs  in LangSmith API. Please confirm your LANGCHAIN_ENDPOINT. ConnectionError(MaxRetryError("HTTPConnectionPool(host='localhost', port=1984): Max retries exceeded with url: /runs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x2d0c00d90>: Failed to establish a new connection: [Errno 61] Connection refused'))"))
Connection error caused failure to patch http://localhost:1984/runs/8703dda5-af15-4f7c-adeb-9636a07f847b  in LangSmith API. Please confirm your LANGCHAIN_ENDPOINT. ConnectionError(MaxRetryError("HTTPConnectionPool(host='localhost', port=1984): Max retries exceeded with url: /runs/8703dda5-af15-4f7c-adeb-9636a07f847b (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x2d0bdd640>: Failed to establish a new connection: [Errno 61] Connection refused'))"))
```

How can I disable LangSmith to prevent these?

unfortunatly there is no fix as of now and it is an annoying set of error messages. We will change that shortly

for now, you can disable the logs if you want 
```py
import logging
logging.getLogger('langchain.callbacks.tracers.langchain').setLevel(logging.WARNING)
logging.getLogger('urllib3.connectionpool').setLevel(logging.ERROR)
```

will put a long term solution in the mean time

Thanks a lot! 🙏🏻

I also faced the same issue and using the given suggestion to disable the log for now. Thanks @jjmachan 

applogies for this annoyance @ranjanritesh89 but we will also have a permanent fix with #394 too
Repository: explodinggradients/ragas
Issue: Testset generator not preserving persona and scenario metadata in generated samples
Root cause: Unimplemented Feature Gap

When using TestsetGenerator with multiple personas, there's no reliable way to track which persona was used to generate each sample in the resulting testset. The persona information is lost after generation, making it impossible to:
- Correlate generated questions with their source personas
- Verify persona distribution in the generated dataset
- Validate that questions are relevant to their intended personas

Users currently resort to workarounds like string prefixing or processing one persona at a time, which is inefficient and degrades quality. A recommended approach to preserve and access persona metadata in generated samples is needed.

I've created PR https://github.com/explodinggradients/ragas/pull/2389 that adds metadata preservation for SingleHop query generation. The PR adds:
- `persona_name`
- `query_style`
- `query_length`

These fields are automatically populated during sample generation in `SingleHopQuerySynthesizer`, solving the metadata tracking issue for single-hop scenarios. Once merged, it will allow proper tracking and analysis of persona distribution in generated samples.

I hope it is useful!