zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: ToolCallAccuracy returns score higher than 1.0
Root cause: Algorithmic Error

- [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
If multiple tool calls match the reference call, the returned score can be higher than `1.0`.

This can happen in various scenarios, e.g.:
* The tool call returned error message. The agent retried the tool call again (with the same args). As a result, there are multiple tool call entries recorded in the sample.
* The number of tool calls (of the same tool) is up to the agent to decide (can be one, two or more). The `ToolCallAccuracy` is used to ensure at least one call has been made (see code example below).

Ragas version: 0.2.15
Python version: 3.13.2

**Code to Reproduce**
```
    messages = [
        HumanMessage(content="What's the weather in Shanghai at different time of the day tomorrow?"),
        AIMessage(content="...", tool_calls=[
            ToolCall(name="weather_check", args={"location": "Shanghai", "time": "9:00"}),
            ToolCall(name="weather_check", args={"location": "Shanghai", "time": "13:00"}),
            ToolCall(name="weather_check", args={"location": "Shanghai", "time": "21:00"}),
        ]),
    ]

    sample = MultiTurnSample(
        user_input=messages,
        reference_tool_calls=[
            ToolCall(name="weather_check", args={"location": "Shanghai"})
        ]
    )

    scorer = ToolCallAccuracy()
    result = asyncio.run(scorer.multi_turn_ascore(sample))

    assert 0.0 <= result <= 1.0, "Score should be between 0.0 and 1.0"
    # Assertion fails because the result is 3.0
```

**Expected behavior**
ToolCallAccuracy returns 1.0 if there is a match (one or more) for the reference tool call.

According to [the documentation](https://docs.ragas.io/en/v0.1.21/references/metrics.html), the score must never be above 1.0:
> Score can range from 0 to 1 with 1 being the best.

Looked at the [code](https://github.com/explodinggradients/ragas/blob/08953bfcfe14dc33a9cd5962c436a273586548c8/ragas/src/ragas/metrics/_tool_call_accuracy.py#L90) for the tool call accuracy

<img width="530" alt="Image" src="https://github.com/user-attachments/assets/a478eb2e-65be-4c20-a70f-9dc222dccb82" />

the implementation is running a nested loops over reference_tool_call and pred_tool_calls. So in a sense it is checking all combinations. Ideally zip should have been used something like

```
for ref_tool_call, pred_tool_call in zip(reference_tool_calls, pred_tool_calls):
      if ref_tool_call.name == pred_tool_call.name:
          arg_score = await self._get_arg_score(
              pred_tool_call.args, ref_tool_call.args, callbacks
          )
          score += arg_score
```
Repository: explodinggradients/ragas
Issue: test for cross_encoders fails when False
Root cause: Validation Gap

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**

When initializing `HuggingfaceEmbeddings` with `sentence-transformers/all-mpnet-base-v2`  I get the following error:

```
pydantic_core._pydantic_core.ValidationError: 1 validation error for HuggingfaceEmbeddings
  Value error, The truth value of an empty array is ambiguous. Use `array.size > 0` to check that an array is not empty. [type=value_error, input_value=ArgsKwargs((), {'model_na...ers/all-mpnet-base-v2'}), input_type=ArgsKwargs]
    For further information visit https://errors.pydantic.dev/2.11/v/value_error
```
this happens because of the following lines in the `__post_init__` method

```
self.is_cross_encoder = bool(
    np.intersect1d(
        list(MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING_NAMES.values()),
        config.architectures,
    )
)
```
an empty array (which results when intersection is empty) is not amenable to boolean evaluation. 

A fix would check for the size of the array not being 0, e.g.

```
self.is_cross_encoder = np.intersect1d(
        list(MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING_NAMES.values()),
        config.architectures,
    ).size != 0
```

I can submit a PR with this.


Hi @ryszardtuora ,

I’ve submitted a PR that fixes this issue: #2308. ✅
The fix ensures that HuggingfaceEmbeddings handles empty arrays safely, and all relevant tests pass. The CI pipeline is green.

Cc: @jjmachan  for visibility.

Please let me know if any further changes are needed.

Thanks!
Repository: explodinggradients/ragas
Issue: Incorrect usage of await llm.generate(...) instead of await llm.agenerate(...) in async context
Root cause: External Dependency Breakage

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

Summary

In recent versions of Langchain, llm.generate(...) is now a synchronous method, while llm.agenerate(...) is the asynchronous alternative. However, in the current Ragas codebase (e.g., pydantic_prompt.py), I see await llm.generate(...) being used.

This results in a runtime error, because llm.generate() is not a coroutine and cannot be awaited.


Ragas version: 0.2.15
Python version: 3.11
Langchain version: >=0.2.43

**Code to Reproduce**
	1.	Install Ragas 0.2.15 and Langchain 0.2.43+
	2.	Attempt to evaluate using a Langchain LLM
	3.	Ragas internally awaits llm.generate(...) — leads to runtime error

**Error trace**
TypeError: object LLMResult can't be used in 'await' expression


**Expected behavior**
Either 
await llm.agenerate(...)
Or
llm.generate(...)

If compatibility with older Langchain versions is needed, a conditional fallback could be added:
if hasattr(llm, "agenerate"):
    result = await llm.agenerate(prompts)
else:
    result = llm.generate(prompts)

**Additional context**
N/A

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->
Repository: explodinggradients/ragas
Issue: Answer_relevance value negative issue.
Root cause: Validation Gap

[ O ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
The answer_relevance value is coming out negative. 
ex. -0.001454476865077159

Ragas version: 0.2.15
Python version: 3.11.11

**Code to Reproduce**
It probably won’t be necessary.
In ragas/ragas/src/ragas/metrics/_answer_relevance.py → the ResponseRelevancePrompt class may have produced a negative value via the LLM. 
It looks like you need to add post-processing logic or strengthen the prompt.

**Error trace**
-
**Expected behavior**
The answer_relevance value should always output only 0 or 1.

**Additional context**
-


Hi @InhyeH, 

It would be great if you could share the specific example where the score came out to be negative.

One possible reason for a negative score could be the use of similarity metrics — for instance, if cosine similarity is being used, its range is from -1 to 1, so negative values are entirely possible when the vectors are dissimilar or even opposed in direction.

That said, I’d love to take a closer look at the exact input where this happened. It’ll help confirm whether it’s indeed due to the similarity computation or if there’s something else going on.
Repository: explodinggradients/ragas
Issue: Add support to evaluating conversation chains
Root cause: Unimplemented Feature Gap

Currently, ragas can evaluate only single-shot RAG pipelines, ie <Question, context, answer>.  In chat-based use cases the question is composed of chat history. For example

> What can you tell me about Gary Cherone?
>Answer from bot
> Did he sing well?

One way to deal with this is to transform the chat history to compose it into a single question to evaluate. [Reference](https://gpt-index.readthedocs.io/en/stable/core_modules/query_modules/chat_engines/usage_pattern.html#low-level-composition-api)
Repository: explodinggradients/ragas
Issue: The output answer is the same as the reference answer, and the blue score has always been 0
Root cause: Configuration Error

The output answer is the same as the reference answer, and the blue score has always been 0 

demo： 

if __name__ == '__main__':
    query = "今天天气怎么样"

    reference = """今天天气很不错"""
    response ="""今天天气很不错"""

    metrics = [BleuScore()]

    result = evaluate(
        dataset=dataset,
        metrics=metrics,
        llm=llm,
        embeddings=embeddings
    )

    print(result)

Hi @LillyChen, 

Thanks for raising this!

Under the hood, the BLEU metric in Ragas uses the sacrebleu library. By default, sacrebleu uses a tokenizer that’s not optimized for Mandarin, which can lead to a BLEU score of 0 even when the output and reference are identical.

Since you’re evaluating Chinese text, you should explicitly set tokenize='zh' in the BLEU calculation. This tells sacrebleu to use a Chinese-specific tokenizer that better handles character boundaries.


```
from ragas.dataset_schema import SingleTurnSample, EvaluationDataset
from ragas import evaluate
from ragas.metrics import BleuScore

reference = "今天天气很不错"
response ="今天天气很不错"


dataset = EvaluationDataset(samples=[SingleTurnSample(response=response, reference=reference)])
metrics = [BleuScore(kwargs={"tokenize": "zh"})]

result = evaluate(
    dataset=dataset,
    metrics=metrics,
)

print(result)
```
Ouput
```
Evaluating: 100%|██████████| 1/1 [00:00<00:00, 232.56it/s]
{'bleu_score': 1.0000}
```


I encountered the same problem when calculating RougeScore for Chinese. How should I handle it
Repository: explodinggradients/ragas
Issue: function `generate_with_langchain_docs()` and the evaluation function `evaluate()` do not support the caller actively stopping the task execution.
Root cause: Unimplemented Feature Gap

**Describe the Feature**
There is an issue where the evaluation set generation function `generate_with_langchain_docs()` and the evaluation function `evaluate()` are both time-consuming and synchronous during runtime. They do not support the caller actively stopping the task execution.

**Why is the feature important for you?**
In a production environment, it is even more critical for tasks to support active termination.




Thanks for raising the feature request @lxhappy110 

It'd be great to have cancellable task.
Repository: explodinggradients/ragas
Issue: How can I generate qa samples in parallel or concurrently to access them one by one when generating a big batch?
GitHub labels: question

module-testsetgen

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
How can I generate qa samples in parallel or concurrently to access them one by one when generating a big batch?

**Additional context**
I've tried generating qa samples with sample count set to 1 and start multiple generations. But this conlcudes in very similar qa samples in comparison to generating all of them in a single call to the generation method.
Repository: explodinggradients/ragas
Issue: Allow overriding `PydanticPrompt`'s instruction and examples in the `__init__` method by changing them to instance attributes with fallback
Root cause: Architectural Constraint

**Describe the Feature**
I expect to be able to initialize a subclass of `PydanticPrompt` and overriding the examples/instruction in the `__init__` directly.

i.e.
```python
ContextRecallClassificationPrompt(instruction="...", examples=[...])
```

**Why is the feature important for you?**
I am doing some evals and using `LLMContextRecall` / other related `MetrciWithLLM` classes, they provide a default prompt which is convenient, but overriding the prompt is not so convenient. For example, `ContextRecallClassificationPrompt` is the prompt class used in `LLMContextRecall`, and its `__init__`  method does not allow changing the examples (because they are class attributes), so the only way to go around it is to subclass `ContextRecallClassificationPrompt` and change the examples, or to initialize an instance of `ContextRecallClassificationPrompt` and then setting the instruction/examples (`prompt.examples = examples`). Both are more verbose than I would expect, and feels like ragas is making things harder than necessary.

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Closing this as obsolete.

- New metrics use function-based prompts, not `PydanticPrompt` classes
- You can customize by modifying prompt functions directly
- The pain point of "overriding class attributes in init" no longer exists
Repository: explodinggradients/ragas
Issue: [Feature Request] Add support for LangChain Bedrock LLMs and embedding models
Root cause: Unimplemented Feature Gap

Could you please add support to pass in a different LLM and embedding model?

```python
import boto3
from botocore.config import Config
from langchain.chat_models import BedrockChat
from langchain.embeddings import BedrockEmbeddings

my_config = Config(retries={'max_attempts': 10, 'mode': 'standard'})
bedrock_inference = boto3.client(service_name='bedrock-runtime', config=my_config)

llm = BedrockChat(
    client=bedrock_inference,
    model_id="anthropic.claude-v2",
    model_kwargs={"temperature": 0.0},
)

embeddings = BedrockEmbeddings(
    client=bedrock_inference,
    model_id='amazon.titan-embed-text-v1',
)
```


hey @austinmw thanks for raising this issue - I will take a look at this and make a PR for adding the same. 
currently there is a bug in the way we are handling temperature so you can't use it out of the box

also thank you so much for taking the time to write out the code as well, that really helped a lot 🙂 
will get this out shortly

Hey @jjmachan , just curious what the status is on this? I noticed a v0.0.18 release with #189 that seems to provide some fixes, but I still get `OpenAIKeyNotFound` when I try:

```python
answer_rel_chain = RagasEvaluatorChain(metric=answer_relevancy)
```

Also if you need help testing Bedrock models I'm happy to help!
Repository: explodinggradients/ragas
Issue: `LangchainLLMWrapper.generate_text` does not reset temperature correctly
Root cause: Algorithmic Error

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Both `LangchainLLMWrapper.generate_text` and `LangchainLLMWrapper.agenerate_text` sets the inner langchain_llm's temperature to the method's input afterwards

Ragas version: 0.2.15
Python version: 3.13.3

**Code to Reproduce**
https://github.com/explodinggradients/ragas/blob/0773595180ce9768af1bab5508c63e5be2ca640a/ragas/src/ragas/llms/base.py#L195-L233

`old_temperature` does not actually store the original temperature of the inner `langchain_llm`, it is storing the result of `temperature or self.get_temperature(n=n)`.

Also, why is `self.get_temperature(n=n)` used when `temperature is None`, instead of respecting the setting of `langchain_llm` and use that instead?

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


This issue has been successfully fixed in the current version of Ragas (PR #2295)

Temperature is now properly restored to its original value after method execution. Both `generate_text()` and `agenerate_text()` correctly save and restore the original temperature setting.
Repository: explodinggradients/ragas
Issue: PyPI page for 3.x is worse than 2.x
Root cause: Configuration Error

[X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Comparing https://pypi.org/project/ragas/0.2.15/ to https://pypi.org/project/ragas/0.3.0/ - the 0.3.0 page has lost the readme and the packages no longer include the license file. It would also be good to link to the homepage and indicate the license in the pypi packaging.

Ragas version:
Python version:


All license checking tools are currently flagging the new version of Ragas as having an unknown license because it is missing from PyPI.

Hey @hyandell @VelizarVESSELINOV we have fixed this in the new 0.3.2 release

Thank you for reporting this and hope everything is fixed

closing this for now but feel free to comment below if stuff still breaks
Repository: explodinggradients/ragas
Issue: Is there an metric  to measure the alignment between the Retrieved Contexts and the ground truth?
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Is there an metric to measure the alignment between the Retrieved Contexts and the ground truth?

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us?
Repository: explodinggradients/ragas
Issue: Getting sharply different values for results, using llama_index example under integrations
Root cause: Documentation Deficiency

Tried to run the llama_index example from integrations folder under docs - 

Got the following result:
`{'ragas_score': 0.0779, 'faithfulness': 0.9600, 'answer_relevancy': 0.8940, 'context_recall': 0.6800, 'context_precision': 0.0210, 'harmfulness': 0.0000}`

Which seems to be way off compared to the results in the notebook.
`{'ragas_score': 0.5142, 'faithfulness': 0.7000, 'answer_relevancy': 0.9550, 'context_precision': 0.2335, 'context_recall': 0.9800, 'harmfulness': 0.0000}`

For reference, I am using an Azure Open AI Instance, with GPT4 as my model. 

Is this normal/expected? 


Hey @vishhvak , thanks for asking. This is normal as we had updated metrics/datasets multiple times after the notebook had the last run. 
I think it would be nice to add a caution at the end of tutorial notebooks mentioning this as well. Let me know if you like to work on this :)
Repository: explodinggradients/ragas
Issue: Support for Batch API
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Is it possible to support Batch API for Ragas?

**Why is the feature important for you?**
Batch API has lower cost, so it'll lower the evaluation cost for LLM-based metrics


Thanks for raising this @minhrongcon2000 

This'll be a good feature to have.
Repository: explodinggradients/ragas
Issue: Brpken links to HOWTO metrics tracing
GitHub labels: documentation

enhancement

**Describe the Feature**
Link broken on the documentation


**Additional context**
In the 3rd FAQ in:
https://docs.ragas.io/en/stable/#frequently-asked-questions
There is a broken link pointing to:
https://docs.ragas.io/howtos/customizations/metrics/tracing/

I think it should point to here instead:
https://docs.ragas.io/en/stable/howtos/customizations/metrics/tracing/


<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Hi @ulisesrey , I’ve submitted a PR to fix the broken link in the FAQ section: #2238. 
The link now points to the correct path in the stable docs. 
Thanks for reporting this!
Repository: explodinggradients/ragas
Issue: progress is stuck (or maybe too long) for generate_with_langchain_docs !!!
Root cause: Resource Mishandling

# 🚨 Bug Report: progress is stuck (or maybe too long) for `generate_with_langchain_docs`

- [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

---

### 📄 Describe the bug

The execution of `generate_with_langchain_docs()` is stuck at 50% with no progress for over an hour. No error is thrown, and CPU usage is minimal. It seems like the generation process either hangs or is too long for practical usage.

---
### 🐍 Python version 
Python 3.132.2

### 💻 Ragas version
Ragas 0.3.0

---

### 📌 Code to Reproduce

```python
from ragas.testset import TestsetGenerator
from pandas import DataFrame, concat
from PDF_load.pdf_loader import get_content
from LLM.settings import generator_llm, generator_embeddings
import pathlib
import asyncio

# Load PDF content
docs = get_content()

# Create generator
generator = TestsetGenerator(
    llm=generator_llm, embedding_model=generator_embeddings
)

output_dir = pathlib.Path("Output")
output_dir.mkdir(exist_ok=True)

async def run():
    docs = get_content()
    for item in docs:
        file_name = item.get("file_name")
        doc = item.get("doc")

        dataset = generator.generate_with_langchain_docs(doc, testset_size=10)
        df = dataset.to_pandas()

        output_path = output_dir / f"{file_name.split('.')[0]}.csv"
        df.to_csv(output_path, index=False)

asyncio.run(run())
```
---
Generating Scenarios:  50%|████████████████                | 1/2 [27:14<27:14, 1634.71s/it]
Repository: explodinggradients/ragas
Issue: How to use local embedding？
Root cause: Documentation Deficiency

**How to use local embedding？** 

I want to use my local embedding api which is compatible with HuggingfaceEmbeddings？what should i do

For running local embeddings, the main pitfall usually falls under Problem Map No 5 (semantic ≠ embedding) — the wrapper expects HuggingFace’s exact interface, but your local API may expose slightly different input/output semantics.

Practical step: make sure your local endpoint returns the same JSON field structure and dimensions that HuggingfaceEmbeddings would, otherwise the call fails even though the server is reachable. A quick test is to log one embedding vector and compare shape + field names against HF’s.

If you’d like, I can share the broader checklist of common failure modes (16 in total) — it’s a semantic firewall approach and doesn’t require changing infra. Just let me know.

It's possible already I believe.

```py
from ragas.embeddings import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(
    model="sentence-transformers/all-mpnet-base-v2",
    device="cuda"  # or "cpu"
)

```

This runs embeddings locally without any API calls.

You can also use `LiteLLM` or write a custom embedding using `BaseRagasEmbedding`

It seems the issue was answered, closing this now.