zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: ContextRelevanceScore for same user query and context varies in a Jupyter Notebook and a web app
Root cause: Validation Gap

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
A clear and concise description of what the bug is.
Something is off with ragas context relevance (from ragas.metrics import ContextRelevance). When I test it in a Jupyter notebook using a query and retrieved context, I get a context relevance score of 0 which is correct.

But when I run the same through a web app, for the same query and retrieved context, I get a context relevance score of 1.

How do I debug this? The score should be 0.

I am using the version 0.2.15 in both the web app and jupyter notebook.
I also tried hard-coding the code from Jupyter notebook in the web app and I still got a score of 1.

Ragas version:0.2.15
Python version:3.10.16

**Code to Reproduce**
Share code to reproduce the issue
I am using OpenAI gpt-4o model.
Retrieved context is a list of str. I cannot paste the values due to confidentiality.
```
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import ContextRelevance

sample = SingleTurnSample(
     user_input="Where is the pectin OG 903 CS produced?",
        retrieved_contexts=retrieved_contexts
)

scorer = ContextRelevance(llm=evaluator_llm)
score = await scorer.single_turn_ascore(sample)
print(score)
```

**Error trace**
There is no error. It is a bug.

**Expected behavior**
A clear and concise description of what you expected to happen.
The web app should return a ContextRelevanceScore of 0 just like the Jupyter Notebook.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Please ignore. I found the issue. The casing in the query was not the same which resulted in different scores.

"Where is the pectin OG 903 CS produced?" query in notebook
"Where is the pectin og 903 cs produced?" query in the app

It is quite amazing how such small changes can impact how the LLM thinks.
Repository: explodinggradients/ragas
Issue: Can not use custom embedding model
Root cause: Interface Contract Mismatch

I followed the instructions and used doubao embedding model  with LangchainEmbeddingsWrapper, but an error occurred

(<class 'openai.BadRequestError'>, BadRequestError("Error code: 400 - {'error': {'code': 'InvalidParameter', 'message': 'The parameter `input[0]` specified in the request are not valid: expected a string, but got `[4599 574 279 1176 2307 19763 30]` instead. Request id: 021754361159016ee5eb03f9031c9451997bdb14cb7183a6316d9', 'param': 'input[0]', 'type': 'BadRequest'}}"), <traceback object at 0x134f2a300>)

here is the code

```
def do():
    evaluator_llm = ChatOpenAI(
        openai_api_base="https://ark.cn-beijing.volces.com/api/v3",
        openai_api_key="xxxx",  # app_key
        model_name="ep-20241017183117-br5f4",  # 推理接入点
    )
    evaluator = LangchainLLMWrapper(evaluator_llm)

    # embedding_model = HuggingFaceEmbeddings(model_name="bert-base-uncased")
    embedding_model_doubao = OpenAIEmbeddings(
        base_url="https://ark.cn-beijing.volces.com/api/v3",
        api_key="xxx",  # app_key
        model="ep-20250805102725-7m5n4",  # 推理接入点
    )
    embedding = LangchainEmbeddingsWrapper(embedding_model_doubao)

    sample = SingleTurnSample(
        user_input="When was the first super bowl?",
        response="The first superbowl was held on Jan 15, 1967",
        retrieved_contexts=[
            "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles."
        ]
    )
    scorer = ResponseRelevancy(llm=evaluator, embeddings=embedding)
    result = scorer.single_turn_score(sample)
    print(result)
```

This error matches Problem Map No 2 (Interpretation Collapse) — the request payload type doesn’t align with what the embedding API expects. In your case, the wrapper is handing off a list where a string is required, so the model rejects it even though the config looks fine.

Quick check: log the raw input before it’s sent, and confirm it’s flattened into a single string per call. That usually resolves this mismatch.

If you’d like, I can share the full set of mapped failure modes + fixes (semantic firewall style, no infra changes). Just let me know.

The wrapper is NOT the problem - it correctly passes strings to the underlying LangChain OpenAIEmbeddings object.

The error message clearly shows:
expected a string, but got `[4599 574 279 1176 2307 19763 30]`

The Doubao API received a tokenized list (token IDs) instead of raw text strings.

Try using ragas's modern `OpenAIEmbeddings` provider instead of `LangchainEmbeddingsWrapper`.


It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Feature: Optional Handit.ai integration to auto-fix prompts after Ragas evaluations
GitHub labels: enhancement

# Feature: Optional Handit.ai integration to auto-fix prompts after Ragas evaluations

Hi team 👋

Love what you’ve built with Ragas — it’s become a go-to tool for evaluating LLM apps.

We’ve been thinking about an optional feature that could make Ragas evaluations even more actionable: integrating **[Handit.ai](https://handit.ai)**, our open-source “autonomous engineer” that monitors and fixes AI 24/7.

---

## Problem / Opportunity
Ragas gives developers great evaluation metrics, but acting on failed/low-scoring results often means a manual process.  
This slows down iteration and keeps improvements separate from the evaluation workflow.

---

## Proposed Solution
Add an **optional parameter** (e.g., `handit_enabled=True`) that:

1. Sends low-scoring/failed evaluation samples + context to Handit.
2. Handit automatically suggests or applies prompt/agent improvements.
3. Users can review or auto-apply fixes.

> With one extra flag, users could go from **just evaluating** → to **evaluating and fixing** automatically.

---

## Benefits
- Speeds up the evaluation → improvement cycle.
- No extra setup for Ragas users.
- 100% opt-in.
- Integration maintained by the Handit team (no added load for Ragas maintainers).

---

## Next Steps
If the team is open to it, we can prepare a PR adding this as an optional enhancement.

What do you think?

— **Cristhian** @ [Handit.ai](https://handit.ai)


Sounds interesting integration @ccgomezn 

Please check out the newer metrics collections approach using instructor via llm_factory and feel free to open a PR for this integration following that :) 
Would be happy to merge it.
Repository: explodinggradients/ragas
Issue: answer_correctness fails with OutputParserException: expected "text" but got "statements"`
Root cause: Interface Contract Mismatch

- [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**  
When running `evaluate()` with the `answer_correctness` metric, the evaluation fails with an `OutputParserException` caused by a `pydantic.ValidationError`.  
The error occurs because the `statement_generator_prompt` inside `answer_correctness` returns JSON with a `statements` field (e.g., `{"statements": [...]}`) but the output parser is using a model (`StringIO`) that expects a `text` field.  
This mismatch causes the parser to throw a `Field required` error for `text`.

Ragas version: v0.2.15  
Python version: 3.9.x  

**Code to Reproduce (pseudocode - Cannot share full reproduction code as it contains proprietary information)**  
```pseudo
# prepare minimal dataset
create dataset with columns:
  - question: list[str]
  - answer: list[str]
  - ground_truth: list[str]

# initialize metric(s)
metrics = [answer_correctness]  # from ragas.metrics

# (depending on environment) prepare LLM and embeddings
llm = <any langchain-compatible LLM>               # e.g., a local stub or provider-backed
embeddings = <any langchain-compatible embeddings> # shape compatible with ragas

# run evaluation
score = ragas.evaluate(
    dataset,
    metrics=metrics,
    llm=llm,
    embeddings=embeddings
)

# observe failure
# OutputParserException bubbling up from:
#   statement_generator_prompt → PydanticOutputParser(StringIO[text]) → ValidationError (missing "text")
print(score.to_pandas())
```

**Error trace**
```
OutputParserException: Failed to parse StringIO from completion {"statements": ["..."]}.
Got: 1 validation error for StringIO
text
Field required [type=missing, input_value={'statements': [...]}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.11/v/missing
```

**Expected behavior**  
The `statement_generator_prompt` output should be successfully parsed without raising a `ValidationError`. The parser schema and the prompt output format should be aligned.

**Additional context**  
The `statement_generator_prompt` examples in `answer_correctness.get_prompts()` show an expected model with a `statements: List[str]` field, but the parser is still configured for a `StringIO` model with a single `text` field. This mismatch causes the evaluation to fail when the LLM outputs match the examples but not the parser schema.


I got the same error while using NoiseSensitivity!

This issue should be fixed in newer versions of ragas.

Additionally, can you try our new metrics collections (`from ragas.metrics.collections ...`) approach ?


It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: divide by zero encountered in divide
Root cause: Validation Gap

When it tries to calculate the "ragas_score" it does the harmonic mean on the metrics, if one of the metrics is equal to 0 by carrying out the calculation on evaluate:
"reciprocal_sum = np.sum(1.0 / np.array(values))" occurs: "RuntimeWarning: divide by zero encountered in divide"
therefore the ragas_score is set to 0.

@gdelpuente , Nice catch! This can be solved by #218
Repository: explodinggradients/ragas
Issue: Migrate workspace.toml to pyproject.toml
Root cause: Architectural Constraint

I think we should now remove workspace.toml file since this is now not a monorepo anymore

_Originally posted by @jjmachan in https://github.com/explodinggradients/ragas/pull/2178#discussion_r2280723723_
            

On it.

Perhaps better to move the ragas out to project root as well.
Repository: explodinggradients/ragas
Issue: "BedrockChat" object has no field "temperature" error when trying to use Bedrock
Root cause: Unimplemented Feature Gap

Hi, I have a test evaluation code below, but when I run it, it keeps throwing an error with trying to get `temperature` from the `BedrockChat` object (this class doesn't have `temperature` field). Am I doing anything wrong? Or can someone give me some directions?

```python
from langchain.chat_models import BedrockChat
from ragas.metrics import (
    context_precision,
    answer_relevancy,
    faithfulness,
    context_recall,
)
from ragas.metrics.critique import harmfulness
from datasets import load_dataset
from ragas import evaluate

chat = BedrockChat(
    model_id="anthropic.claude-instant-v1",
    model_kwargs={"temperature":0.0}
)

fiqa_eval = load_dataset("explodinggradients/fiqa", "ragas_eval")
faithfulness.llm.langchain_llm = chat
result = evaluate(
    fiqa_eval["baseline"].select(range(5)),  # showing only 5 for demonstration
    metrics=[faithfulness],
)
```

Hey @nmklong thanks for pointing this out, Bedrock is not supported at the moment but we will get that out by this weekend

@jjmachan awesome to hear that, thank you!

thankfully #227 will fix this - will release this soon
Repository: explodinggradients/ragas
Issue: Make test e-2e installation component fails due to tock incompatibility with MacOS version
Root cause: Environment Incompatibility

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When trying to run make test e-2e I receive an incompatability error: `torch==2.8.0 @ registry+https://pypi.org/simple` can't be installed because it doesn't have a source distribution or wheel for the current platform

I was however able to run it in a docker container instead.

Ragas version:  Ragas 
Python version: Python 3.13

**Code to Reproduce**
Run make teste-2e on a macosx_13_0_x86_64.

**Error trace**

```
error: Distribution `torch==2.8.0 @ registry+https://pypi.org/simple` can't be installed because it doesn't have a source distribution or wheel for the current platform

hint: You're on macOS (`macosx_13_0_x86_64`), but `torch` (v2.8.0) only has wheels for the following platforms: `manylinux_2_28_aarch64`, `manylinux_2_28_x86_64`, `macosx_11_0_arm64`, `win_amd64`; consider adding your platform to `tool.uv.required-environments` to ensure uv resolves to a version with compatible wheels
```

**Expected behavior**
e2e test runs without issue 
OR if the dependancy can't beresolved
provide instructions and docker files to run in docker container instead.



This should have also been fixed in PR #2222 
Could you please and confirm?

let us know if this it was fixed @mnedelko 

Closing as stale
Repository: explodinggradients/ragas
Issue: TypeError: BaseChatModel.generate() missing 1 required positional argument: 'messages'
Root cause: Interface Contract Mismatch

When I generate a test dataset using the code provided in the document, this error occurs.
The code:
### Add custom llms and embeddings
generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
critic_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
embeddings_model = OpenAIEmbeddings()

### Change resulting question type distribution
testset_distribution = {
    "simple": 0.25,
    "reasoning": 0.5,
    "multi_context": 0.0,
    "conditional": 0.25,
}

### percentage of conversational question
chat_qa = 0.5


test_generator = TestsetGenerator(
    generator_llm=generator_llm,
    critic_llm=critic_llm,
    embeddings_model=embeddings_model,
    testset_distribution=testset_distribution,
    chat_qa=chat_qa,
)

testset = test_generator.generate(documents, test_size=5)

The error detail：
File d:\ProgramData\Anaconda3\envs\xx\lib\site-packages\ragas\testset\testset_generator.py:179, in TestsetGenerator._filter_context(self, context)
    [177](file:///d%3A/ProgramData/Anaconda3/envs/seckb/lib/site-packages/ragas/testset/testset_generator.py?line=176) human_prompt = SCORE_CONTEXT.format(context=context)
    [178](file:///d%3A/ProgramData/Anaconda3/envs/seckb/lib/site-packages/ragas/testset/testset_generator.py?line=177) prompt = ChatPromptTemplate.from_messages([human_prompt])
--> [179](file:///d%3A/ProgramData/Anaconda3/envs/seckb/lib/site-packages/ragas/testset/testset_generator.py?line=178) results = self.critic_llm.generate(prompts=[prompt])
    [180](file:///d%3A/ProgramData/Anaconda3/envs/seckb/lib/site-packages/ragas/testset/testset_generator.py?line=179) output = results.generations[0][0].text.strip()
    [181](file:///d%3A/ProgramData/Anaconda3/envs/seckb/lib/site-packages/ragas/testset/testset_generator.py?line=180) score = load_as_score(output)

TypeError: BaseChatModel.generate() missing 1 required positional argument: 'messages'

Who can help me solve this problem?

Hi @zjucsgk ,apologies for late reply. Can you help us with the ragas & python version you're using?

applogies from my side for the delay, this is probably because your passing a Langchain LLM where a `RagasLLM` is expected.

how you should call is like 
```python
from langchain.chat_models import ChatOpenAI
from ragas.llms import LangchainLLM

generator_llm = LangchainLLM(llm=ChatOpenAI(model=openai_generator_llm))
critic_llm = LangchainLLM(llm=ChatOpenAI(model=openai_filter_llm))
embeddings_model = OpenAIEmbeddings()  # type: ignore
```

do try it out @zjucsgk and let us know if its fixed

apologies again for the delay but hope this fixes it and welcome to the community :)

> Hi @zjucsgk ,apologies for late reply. Can you help us with the ragas & python version you're using?
@shahules786 Thanks
Python 3.10.13
ragas-0.0.18 

@jjmachan OK，its fixed. Thank you
Repository: explodinggradients/ragas
Issue: Is Synthetic testset generation only possible through knowledge graph generation or is it possible to generate dataset without Knowledge graph as well?
Root cause: Unimplemented Feature Gap

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

My question is:
Is it a possibility to generate Synthetic testset without knowledge graph generation? Reason is KG generation is complex and time consuming, it would be nice to know if we can skip this process and generate straightforward questions just by looking at the document?

If I'm not wrong, the earlier ragas had this way of generating synthetic dataset(without using Knowledge graph). 


hey @saranyashalya so currently it is not. the KG is required although we are trying to figure out how we can get past it too.

> it would be nice to know if we can skip this process and generate straightforward questions just by looking at the document?

actually we do exactly this with the single hop synthesizer, have you checked that out?

also would love your feedback on https://github.com/explodinggradients/ragas/issues/2231 too if you have some time 🙂 

Thanks for your response! I see that single hop synthesizer is also based on KG - src/ragas/testset/synthesizers/single_hop/specific.py. Am I missing something? 

I have provided my response for #2231 .

Currently, I am looking for methods to generate test cases. The process of building a knowledge graph takes a relatively long time, and I am also having difficulty understanding the relationships between the attributes. Do you have any suggestions?
Repository: explodinggradients/ragas
Issue: Have noticed ragas needs cohere and llama_index libraries to be installed before importing ragas
Root cause: Validation Gap

Not a bug as such :)
I was trying to run this on Google collab and noticed it requires me to install cohere and llama_index before installing ragas for being able to import ragas successfully.  

```
!pip install cohere 
!pip install llama_index
!pip install ragas 

from ragas import evaluate
``` 

Also have noticed its necessary  to set up the openai key before I import ragas, cannot set it after I import ragas

thanks for bringing this up @GautamR-Samagra  I've created #231 to track another problem with dependencies and will close this one with that too

Cheers and welcome to the community :)
Repository: explodinggradients/ragas
Issue: [Feature Request] extract single question scores
Root cause: Unimplemented Feature Gap

Give the possibility to extract from the result also the single question evaluation.

Hi @gdelpuente , thanks for raising the FR. can you explain this a bit more?

when I use the evaluation on a pool of questions the result is one for all for each metric (the average).
I would also like to read the result of the single query

ex:

{
question1 : {context_precision:0,653 ; faithfulness:0,376 ; answer_relevancy....} 
question2:  {context_precision:0,253 ; faithfulness:0,721 ; answer_relevancy...} 
...
final_result: {context_precision:0,876 ; faithfulness:0,345 ; answer_relevancy...} 
}

Hey, have you tried to do `results.to_pandas()` to get sample-level scores?   

hey ask @shahules786 suggested converting it into the dataframe will help you with this. Each row will correspond in the dataframe will currespond to one query and you can do further analysis with just pandas. Do try it out @gdelpuente and tell us if that is what your looking for :)


currently I call evaluate on the individual question and then perform the mean and harmonic mean outside the framework to obtain the summary results.

But the _evaluate_ function internally already does this only that at the end of the cycle it does not display the results calculated for the individual questions.

It could have been an additional feature, also to remain aligned in the formalization of the summary results, to be able to recover the calculated results of the individual questions directly from the framework function

@gdelpuente , understood.  Although you don't have to evaluate the individual questions separately. You can run the entire batch and export results to pandas as mentioned above. Then you can do any operation on the dataframe level. 

but if you want there is a `score_single()` class for each metric that you can use. Its not documented because we might be changing it but you can check the `metrics/base.py` file for more info

I have the same issue, of integrating the ragas metrics into an existing evaluation pipeline which already does other types of evaluation, and I'd like to run it the same way. Thanks for suggesting `score_single()`.

@gdelpuente and @omri374 I'm sharing this notifying you because there is a change for this coming in the proposed `v0.1` that is already merged in

now you can do `score(row: t.Dict)` and `ascore(row: t.Dict)` for the async version. to achieve the same functionality. Please check the code for more info 
https://github.com/explodinggradients/ragas/blob/36a10cba1eb65a501e5a3baed7a23c790594c191/src/ragas/metrics/base.py#L62-L103

closing this for now - fee free to reopen if any issues 🙂
Repository: explodinggradients/ragas
Issue: [Feature Request] use evaluate without ground_truths
Root cause: Validation Gap

allow the evaluate function to work without "ground_truths" by providing the other available measures

Hi @gdelpuente , thanks for asking. This is possible with ragas (metrics such as faithfulness), please update to the latest version. 

also @gdelpuente just curious what aspects are you trying to evaluate?

when you do something like
```python
from ragas import evaluate
from datasets import Dataset
import os

os.environ["OPENAI_API_KEY"] = "your-openai-key"
``` 
 if you prepare your dataset in the format **without ground_truths**
```python
# Dataset({
#     features: ['question', 'contexts', 'answer'],
#     num_rows: 25
# })

dataset: Dataset

results = evaluate(dataset)
# ERROR
``` 

the function return an error because it try to evaluate also the _context_recall_
it might be friendly to use if it evaluated what measures it can calculate from the input it receives

I know you can select metrics as parameter
mine was just a suggestion for improvement
also given that the project's starting paper focuses on measurements without ground_truths

understood, thank you for the valuable suggestion. Will fix this shortly 🙂