zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: llm_factory returns InstructorLLM that breaks testset generation
Root cause: Interface Contract Mismatch

[V] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**

After migrating to llm_factory (which now returns an InstructorLLM), the RAGAS testset generation pipeline is broken. The object returned is missing several methods expected by transforms and TestsetGenerator, including:

set_run_config

agenerate_prompt

Ragas version: 0.3.9
Python version: 3.12.12 

**Code to Reproduce** (adapted from documentation)

from ragas.testset.graph import KnowledgeGraph, Node, NodeType
from ragas.testset.transforms import HeadlinesExtractor, apply_transforms
import openai
from ragas.llms import llm_factory
from google.colab import userdata


openai_client = openai.OpenAI(api_key=userdata.get('OpenAI'))
llm_ragas = llm_factory('gpt-4o-mini', client=openai_client)


ragas_nodes = [
    Node(
        type=NodeType.DOCUMENT,
        properties={
            "page_content": "some text",
            "document_metadata": None,
        },
    )
]
kg = KnowledgeGraph(nodes=ragas_nodes)


headline_extractor = HeadlinesExtractor(llm=llm_ragas)
transforms = [headline_extractor]


apply_transforms(kg, transforms=transforms)


**Error trace** (shorten)
AttributeError                            Traceback (most recent call last)
     21     headline_extractor,
     22 ]
---> 23 apply_transforms(kg, transforms=transforms)
 ....
[/usr/local/lib/python3.12/dist-packages/ragas/prompt/pydantic_prompt.py](https://localhost:8080/#) in generate_multiple(self, llm, data, n, temperature, stop, callbacks, retries_left)
    236             # LangChain doesn't support n parameter directly, so we batch multiple prompts
    237             prompts = t.cast(t.List[t.Any], [prompt_value for _ in range(n)])
--> 238             resp = await langchain_llm.agenerate_prompt(
    239                 prompts,
    240                 stop=stop,

AttributeError: 'InstructorLLM' object has no attribute 'agenerate_prompt'

**Expected behavior**

Normal functioning of HeadlineExtractor.

**Additional context**

Generation using TestsetGenerator seems also broken, returning

AttributeError: 'InstructorLLM' object has no attribute 'set_run_config'

EDIT: this can be avoided using a llamaindex/langchaing wrapper, but wrapper's are deprecated.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Thank you for reporting this issue @simonebmalerba 

Handled it in #2451
Repository: explodinggradients/ragas
Issue: Dependency on GitPython
Root cause: Architectural Constraint

While developing a program that includes ragas in Docker, I encountered an error indicating that git is not available, which led me to discover the dependency on GitPython.
I have two questions regarding this:

What is GitPython used for in ragas?
For use cases like mine (running in Docker without git), would it be safe to suppress this issue using the following approach?

os.environ['GIT_PYTHON_REFRESH'] = 'quiet'

@HarugumoFM The git module is imported and used in the `version_experiment()` function.

But you're right at calling this out. Will take another look at the use and restructure.
Repository: explodinggradients/ragas
Issue: Do the new collection metrics (e.g., AnswerRelevancy) support language adaptation?
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I am migrating to the new ragas.metrics.collections and I noticed that the documentation for [Language Adaptation](https://docs.ragas.io/en/stable/howtos/customizations/metrics/_metrics_language_adaptation/) seems to primarily address the older/standard metrics using the adapt() function.

I would like to know the recommended way to handle prompt language adjustment for the new collection metrics.


@a5512167086 
Currently you can use like:

```py
metric = AnswerCorrectness(llm=llm, reference_llm=ref_llm)

metric.prompt = await metric.prompt.adapt(
    target_language="hindi",
    llm=llm,
    ...
)
```

Additionally, PR #2465 will add save/load capabilities to the new prompts.

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: maximum context length  
Root cause: Unimplemented Feature Gap

Hi There,

Is it possible to add an option to limit the token or word counts with a parameter, or just skip a request with this error instead of breaking all evaluation process?

evaluating with [answer_relevancy]

---------------------------------------------------------------------------
InvalidRequestError                       Traceback (most recent call last)
[<ipython-input-13-cc8ae8de5cb3>](https://localhost:8080/#) in <cell line: 63>()
     62 
     63 for col in columns_to_evaluate:
---> 64     evaluate_column_and_save(df_9_response, col, evaluate)

22 frames
[/usr/local/lib/python3.10/dist-packages/openai/api_requestor.py](https://localhost:8080/#) in _interpret_response_line(self, rbody, rcode, rheaders, stream)
    773         stream_error = stream and "error" in resp.data
    774         if stream_error or not 200 <= rcode < 300:
--> 775             raise self.handle_error_response(
    776                 rbody, rcode, resp.data, rheaders, stream_error=stream_error
    777             )

InvalidRequestError: This model's maximum context length is 4097 tokens. However, your messages resulted in 4218 tokens. Please reduce the length of the messages.

Hi @aydinmyilmaz , thanks for raising the issue. I think in these cases the ideal behavior is that the result of the particular sample will be NaN. We will work on this, as we are also working on adding async support to improve evaluation speed :) 

`max_tokens` is now available. Try with latest ragas.

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: FactualCorrectness.single_turn_ascore() returns 0.0 with Anthropic via InstructorLLM
Root cause: Interface Contract Mismatch

[X] I have checked the https://docs.ragas.io/ and related resources and couldn't resolve my bug.

  Describe the bug

  When using FactualCorrectness with Anthropic via InstructorLLM (wrapped with instructor.from_anthropic()), single_turn_ascore() always returns 0.0 regardless of input.

  The root cause appears to be internal LangChain callback objects passed to decompose_claims() and verify_claims(). Something in the callback chain breaks with the instructor-wrapped Anthropic
  client, causing silent failures or empty results.

  Workaround: Calling the underlying methods directly with callbacks=None returns correct scores.

  Ragas version: 0.4.0
  Python version: 3.12

  Code to Reproduce
```python
import asyncio
import instructor
from anthropic import AsyncAnthropic
from ragas.llms.base import InstructorLLM
from ragas.metrics import FactualCorrectness
from ragas.dataset_schema import SingleTurnSample

async def main():
    # Setup LLM (per docs)
    client = AsyncAnthropic()
    instructor_client = instructor.from_anthropic(client)
    llm = InstructorLLM(
        client=instructor_client,
        model="claude-sonnet-4-20250514",
        provider="anthropic",
    )

    scorer = FactualCorrectness(llm=llm, mode="recall")

    sample = SingleTurnSample(
        response="The Eiffel Tower is located in Paris, France.",
        reference="The Eiffel Tower is in Paris."
    )

    # BUG: Returns 0.0
    score = await scorer.single_turn_ascore(sample)
    print(f"single_turn_ascore: {score}")  # Always 0.0

    # WORKAROUND: Call underlying methods with callbacks=None
    claims = await scorer.decompose_claims(sample.reference, callbacks=None)
    verifications = await scorer.verify_claims(
        premise=sample.response,
        hypothesis_list=claims,
        callbacks=None
    )
    manual_score = sum(verifications) / len(claims) if claims else 0.0
    print(f"manual calculation: {manual_score}")  # Correct score

asyncio.run(main())
```
  Error trace

  No error - the method returns silently with 0.0 instead of the correct score.

  Expected behavior

  single_turn_ascore() should return the same score as manually calling decompose_claims() + verify_claims() with callbacks=None.

  Additional context

  - Affects any Ragas metric using Anthropic via InstructorLLM when callbacks are involved
  - The issue does not occur when calling the underlying methods directly with callbacks=None
  - Possibly related to how LangChain callbacks interact with the instructor-wrapped client




@florind Pleae use the new metrics collections. 

```py
from ragas.metrics.collections import FactualCorrectness
```

Ref: [docs](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/factual_correctness/#factual-correctness)

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: `llm_factory()` doesn't wrap Anthropic clients with instructor
Root cause: Validation Gap


- [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**

When using `llm_factory("claude-...", provider="anthropic")`, the returned `InstructorLLM` does not have the Anthropic client properly wrapped with `instructor.from_anthropic()`. This causes structured output calls to fail.

Ragas version: 0.4.0
Python version: 3.12

**Code to Reproduce**

```python
import asyncio
from anthropic import AsyncAnthropic
from ragas.llms import llm_factory
from ragas.metrics import FactualCorrectness
from ragas.dataset_schema import SingleTurnSample

async def test():
    client = AsyncAnthropic()
    llm = llm_factory("claude-sonnet-4-20250514", client=client, provider="anthropic")

    scorer = FactualCorrectness(llm=llm, mode="recall")
    sample = SingleTurnSample(
        response="The Eiffel Tower is in Paris, France.",
        reference="The Eiffel Tower is located in Paris."
    )

    score = await scorer.single_turn_ascore(sample)  # Fails

asyncio.run(test())
```

**Error trace**

```
TypeError: AsyncMessages.create() got an unexpected keyword argument 'response_model'
```

Full traceback shows `llm_factory` wraps the client but uses OpenAI-style `chat.completions.create()` instead of Anthropic's `messages.create()`.

**Workaround:**

```python
import instructor
from anthropic import AsyncAnthropic
from ragas.llms.base import InstructorLLM

# Manually wrap with instructor before passing to InstructorLLM
client = AsyncAnthropic()
instructor_client = instructor.from_anthropic(client)

llm = InstructorLLM(
    client=instructor_client,
    model="claude-sonnet-4-20250514",
    provider="anthropic",
)
```

**Expected behavior**

`llm_factory()` should internally call `instructor.from_anthropic()` when `provider="anthropic"` is specified, so that the returned LLM works out of the box.


Thanks for reporting this @florind 

Please switch to new metrics collections for starters. We've a new `from_provider("anthropic/claude-3-sonnet")` which creates its own client. But since you're passing your client, a fix is needed. Added it in PR #2472
Repository: explodinggradients/ragas
Issue: Makefile uses unsupported --active flag with recent uv versions
Root cause: Environment Incompatibility

***

[ x ] I have checked the [[documentation](https://docs.ragas.io/)](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**  
The Makefile uses `uv run --active` in commands like `format`, `type`, and `check`, but this flag is not supported in recent versions of `uv` . This causes errors such as "unexpected argument '--active' found" when running `make check` or similar commands, even after following the CONTRIBUTING.md instructions.

Ragas version: main branch (latest)  
Python version: 3.13.1

**Code to Reproduce**  
Run these commands after cloning the Ragas repo and following CONTRIBUTING.md:
```bash
make install
make check
```

**Error trace**
```
error: unexpected argument '--active' found

  tip: a similar argument exists: '--native-tls'

Usage: uv run --native-tls

For more information, try '--help'.
make: *** [Makefile:55: format] Error 2
```

**Expected behavior**  
The `make check` command should run successfully without errors, formatting and linting the code as intended.

**Additional context**  
- Removing `--active` from the Makefile resolves the issue and is compatible with modern `uv` versions.

***



@anistark  should I raise a PR for this?

> [@anistark](https://github.com/anistark) should I raise a PR for this?

Sure thing. Please go ahead.

Turns out, the --active flag actually works in the latest version of uv (v0.9.16). I was using an older version (0.5.21), which is why I was getting errors. After upgrading uv, everything works fine and there’s no need to change the Makefile. I should have checked my uv version properly before raising this issue. @anistark , feel free to close this if you think there’s no change needed in the Makefile. For more details, see the discussion in the uv repo: https://github.com/astral-sh/uv/issues/15457.

Thanks @cgaswin
Repository: explodinggradients/ragas
Issue: Support for system prompts in LLM wrappers
Root cause: Unimplemented Feature Gap

**Describe the Feature**
It would be useful to have a possibility to set system prompts for `InstructorLLM` and `LiteLLMStructuredLLM`.

**Why is the feature important for you?**
Some LLMs require a system prompt to work correctly.
Repository: explodinggradients/ragas
Issue: Support for Hugging Face tokenizers in LLM-based extractors
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Currently, text is split into chunks using _tiktoken_ tokenizers when extracting properties from nodes of a knowledge graph. However, _tiktoken_ only provides tokenizers for OpenAI models. It would be useful to support Hugging Face tokenizers so that number of tokens in each chunk is counted correctly during text splitting.

**Why is the feature important for you?**
This feature would make it easier to use open-source LLMs when building knowledge graphs for generating test sets.

**Additional context**
Text splitting: https://github.com/vibrantlabsai/ragas/blob/v0.4.0/src/ragas/testset/transforms/base.py#L231
Repository: explodinggradients/ragas
Issue: Update Google LLM Integration from Deprecated google-generativeai to Official google-genai SDK
Root cause: External Dependency Breakage

**Describe the Feature**
We request that the Ragas codebase, examples, and documentation officially migrate the Google LLM integration from the legacy Python library, `google-generativeai`, to the new, officially supported Google AI SDK, **`google-genai`**.

This migration should ideally cover:
1.  **Codebase:** Updating internal logic within Ragas's Google provider to use `google-genai` imports and client configuration methods.
2.  **Documentation:** Revising all examples, especially in the [Gemini Integration How-To](https://docs.ragas.io/en/latest/howtos/integrations/gemini/), to use the modern `google-genai` package.
3.  **Dependencies:** Checking and updating installation requirements to reflect the current recommended SDK.

**Why is the feature important for you?**
The previously used `google-generativeai` library is a legacy SDK and is officially scheduled for deprecation, with support permanently ending on **November 30, 2025**.

Migrating to `google-genai` is crucial for:

* **Future-proofing:** Ensures Ragas users are relying on a long-term supported dependency.
* **Latest Features:** Guarantees access to the newest Gemini models, improved functionalities (like structured outputs and tool use), and a unified API experience.
* **Preventing Issues:** Avoids future dependency conflicts and maintenance issues that arise from using an unsupported package.

**Additional context**
While Ragas is flexible, current documentation examples still implicitly point towards the deprecated package.

Here is the recommended setup using the modern **`google-genai`** SDK which we request be reflected in the Ragas documentation:

```python
import os
# Recommended: Use the new official SDK
import google.genai as genai 
from ragas.llms import llm_factory

# Configure client using the new SDK methods (e.g., genai.Client)
client = genai.Client(api_key=os.environ.get("GOOGLE_API_KEY"))

# Ragas setup remains the same, using the new client object
llm = llm_factory("gemini-2.5-flash", provider="google", client=client)
```


<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Has been a todo for me this week 😅 

Do you want to open a PR for it? Seems like you've a recommended approach in mind already.
Repository: explodinggradients/ragas
Issue: Testset generation broken: instructor returns empty dict for Dict-type Pydantic models
Root cause: External Dependency Breakage

## Bug Description

Testset generation synthesizers fail because instructor/OpenAI function calling returns `{}` for any Pydantic model containing `Dict` fields.

## Reproduction

```python
from openai import OpenAI
import instructor
from pydantic import BaseModel
import typing as t

client = instructor.from_openai(OpenAI())

class PersonaThemesMapping(BaseModel):
    mapping: t.Dict[str, t.List[str]]

# This always fails - LLM returns arguments='{}'
result = client.chat.completions.create(
    model="gpt-4o-mini",
    response_model=PersonaThemesMapping,
    messages=[{"role": "user", "content": "Map personas to themes"}],
)
```

## Impact

- `SingleHopSpecificQuerySynthesizer` - uses `PersonaThemesMapping` with `Dict[str, List[str]]`
- All synthesizers using Dict-based response models

## Error

```
pydantic_core._pydantic_core.ValidationError: 1 validation error for PersonaThemesMapping
mapping
  Field required [type=missing, input_value={}, input_type=dict]
```

## Notes

- `List[str]` works fine, only `Dict` types fail
- Affects both gpt-4o and gpt-4o-mini
- Discovered while updating language_adaptation.md guide (guide deleted due to this bug)

@sanjeed5 Can you check if PR #2505 fixes it for you? 

@anistark Confirmed! PR #2505 fixes the issue. Tested and approved. 👍
Repository: explodinggradients/ragas
Issue: Customization of test set generation
Root cause: Documentation Deficiency

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I wonder what is the current state regarding customization of the synthetic test set generation? I would like the generated reference answers to contain additional information (like in what chapter of the document the information can be found) Is that somehow easily possible? I checked old issues where it said that you were working on expanding customization for test set generation but i was not sure about the current state




@abcxyz124 You can custom is several ways...

1. Use `set_prompts()` override.
2. Use `llm_context` to add custom context (available since `v0.4.1`) - Probably best for your use-case.



Thank you
Repository: explodinggradients/ragas
Issue: Get metrics evaluation logs
Root cause: Unimplemented Feature Gap

For evaluation monitoring and understanding, it would be great to have the logs of the evaluation process, in particular the llm's responses to produce the final score

For example, for the faithfulness, retrieve the response to LONG_FORM_ANSWER_PROMPT and NLI_STATEMENTS_MESSAGE. 

Possible solutions:
- return another value (a dict, or an object) in addition to the scores with answer logs
- create a class property for each metric, for example `score_logs` updated during evaluation process, and from which we can retrieve logs

I think the following should help with this
1. ironing out the callback bugs
2. documentation on how to log ragas itself effectively

I will be working on both of these shortly but I see you also have a PR ready.
Repository: explodinggradients/ragas
Issue: validation error for PersonaThemesMapping in TestsetGenerator
Root cause: Validation Gap

- [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I met the similar error in https://github.com/vibrantlabsai/ragas/issues/2490
After I adopted the latest fixes, the error was not gone.

Ragas version: 0.4.2.dev9+g4459c3948
Python version: 3.11.9

**Code to Reproduce**
```python
litellm.api_base = settings["endpoint"]
    litellm.api_key = settings["api_key"]
    litellm.api_version = settings["chat_api_version"]
    llm = llm_factory(
        model=f"azure/{settings['chat_deployment']}",
        provider="litellm",
        client=litellm.completion,
        temperature=1.0,
        drop_params=True,
        max_tokens=128000,
    )

    embedding_model = embedding_factory(
        "litellm",
        model=f"azure/{settings['embedding_deployment']}",
        api_base=settings["endpoint"],
        api_key=settings["api_key"],
        api_version=settings["embedding_api_version"],
        drop_params=True,
    )


    persona = [Persona(
        name="New comer",
        role_description="Wants to search from team docs.",
    )]

    generator = TestsetGenerator(llm=cast(BaseRagasLLM, llm), embedding_model=cast(BaseRagasEmbeddings, embedding_model), persona_list=persona)
    generator.generate_with_langchain_docs(documents=documents, testset_size=10);
```

**Error trace**
Not much useful, there is no stack.
```
ERROR:root:Test synthesis failed: <failed_attempts>

<generation number="1">
<exception>
    1 validation error for PersonaThemesMapping
mapping
  Field required [type=missing, input_value={}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
</exception>
<completion>
    ModelResponse(XXXXXXXX)
</completion>
</generation>
```

**Expected behavior**
No exception is thrown.


**Additional context**
Azure OpenAI, tried gpt4o, gpt5, gpt-5-nano.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


I was also having the same issue with GPT-4.1 over Azure OpenAI. The model was returning returns for the PersonaThemesMapping tool call, which causes the Pydantic validation error. What worked for me was using instructor with Mode.JSON instead of the default tool calling:

```python
import instructor
from instructor import Mode
from openai import AsyncAzureOpenAI
from ragas.llms.base import InstructorLLM

azure_client = AsyncAzureOpenAI(
    azure_endpoint=settings["endpoint"],
    api_key=settings["api_key"],
    api_version=settings["chat_api_version"],
)

patched_client = instructor.from_openai(azure_client, mode=Mode.JSON)
llm = InstructorLLM(
    client=patched_client,
    model=settings["chat_deployment"],
    provider="openai",
)
```

I also tried Mode.TOOLS_STRICT first but it was failing too. Hope this helps!