Repository: explodinggradients/ragas
Issue: IndexError occurs when ResponseRelevancy is placed at the top of a metrics list that has multiple items.
Root cause: Resource Mishandling

Hello!

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Bug Overview**
When using `evaluate`, I get IndexError when 
1. ResponseRelevancy is placed at the top of a metrics list like below
2. The metrics list has multiple items.
```python
metrics = [
    ResponseRelevancy(),
    Faithfulness(),
]
```
ResponseRelevancy doesn't calculated and I get NaN.

The error will disappear when I set 
```python
metrics = [
    ResponseRelevancy(),
]
```
or
```python
metrics = [
    Faithfulness(),
    ResponseRelevancy(),
]
```

**Versions**
Ragas version: 0.3.5
Python version: 3.12.11

**Code to Reproduce**
```python
import os
from google.colab import userdata
from langchain_google_genai import ChatGoogleGenerativeAI, GoogleGenerativeAIEmbeddings
from ragas import SingleTurnSample, EvaluationDataset, evaluate
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import ResponseRelevancy, Faithfulness

os.environ["GOOGLE_API_KEY"] = userdata.get("GOOGLE_API_KEY")

# SingleTurnSample from https://docs.ragas.io/en/stable/concepts/components/eval_dataset (slightly modified)
sample1 = SingleTurnSample(
    user_input="What is the capital of Germany?",
    retrieved_contexts=["Berlin is the capital and largest city of Germany."],
    response="The capital of Germany is Berlin.",
    reference="Berlin",
)
sample2 = SingleTurnSample(
    user_input="What's the chemical formula for water?",
    retrieved_contexts=["Carbonated water contains dissolved CO2", "Water has the chemical formula H2O."],
    response="The chemical formula for water is CO2.",
    reference="H2O",
)
dataset = EvaluationDataset(samples=[sample1, sample2])

llm = LangchainLLMWrapper(ChatGoogleGenerativeAI(model="gemini-2.0-flash"))
embeddings = LangchainEmbeddingsWrapper(GoogleGenerativeAIEmbeddings(model="models/gemini-embedding-001"))

metrics = [
    ResponseRelevancy(),
    Faithfulness(),
]
results = evaluate(dataset, metrics, llm=llm, embeddings=embeddings)
results.to_pandas()
```

When I first found this bug, I used OpenAI API. Since I don't have it now, I used Gemini instead.
Feel free to comment out the Gemini-part and use OpenAI API.

As I mentioned earlier, setting `metrics` like below solves this error.
```python
metrics = [
    ResponseRelevancy(),
]
```
or
```python
metrics = [
    Faithfulness(),
    ResponseRelevancy(),
]
```

**Error trace**
Jobs[0] corresponds to ResponseRelevancy of 1st sample.  
Jobs[2] corresponds to ResponseRelevancy of 2nd sample.
```
ERROR:ragas.executor:Exception raised in Job[2]: IndexError(list index out of range)
ERROR:ragas.executor:Exception raised in Job[0]: IndexError(list index out of range)
```

<img width="1132" height="668" alt="Image" src="https://github.com/user-attachments/assets/15ba850a-0ef4-40e0-99e8-8d5395a23921" />

**Expected behavior**
No error occurs and I get correct value.


same with this issue.

I get the same issue with answer relevancy. I assume this might have something to do with concurrent calls. When I put strictness to 1 I do not get this issue.

Give me a little more time for investigation.

The issue is in the concurrent execution system that affects how the first metric receives LLM responses.

As @KenanRustamov mentioned, setting `strictness=1` works.

When multiple metrics run concurrently, the first metric (ResponseRelevancy) may get incomplete LLM responses due to execution timing issues.
So, in isolation, it should work fine cause there's no race condition.

@anistark I made [another PR #2329 ](https://github.com/explodinggradients/ragas/pull/2329) without noticing your work😅

> [@anistark](https://github.com/anistark) I made [another PR #2329 ](https://github.com/explodinggradients/ragas/pull/2329) without noticing your work😅

Ahaa no worries. I reviewed your PR. It would still err in empty case. `resp.generations = []`


@anistark Thank you for reviewing my PR!
I didn't notice that `resp.generations` can be `[]`. To keep up with your thought, could you show me the test case that will make `resp.generations = []`?

> [@anistark](https://github.com/anistark) Thank you for reviewing my PR! I didn't notice that `resp.generations` can be `[]`. To keep up with your thought, could you show me the test case that will make `resp.generations = []`?

Can happen due to multiple cases. Say openai api rate limiting, network timeout, llm return empty due to high overload, content filter or langchain bug and so on.

It's an edge case so pretty hard to happen in an ideal scenario.
However, it's good to handle possible error cases and throw appropriate warnings and errors. 


@anistark Thank you for your explanination! I understood why you wrote `available_generations = len(resp.generations[0]) if resp.generations else 0`. 

Your logging code is more descriptive and helpful for we developers and if-else statement is safer than mine👍 I will close my PR.

By the way, is it OK to keep this issue open until your PR is merged into main branch?

hey @petrov826 thanks for raising the issue and helping us with the initial PR as well 🙂 
we hope we are able to land one of ur PRs soon too!

Also do check out 
[[How to Evaluate a New LLM For Your Use Case](https://docs.ragas.io/en/stable/howtos/applications/benchmark_llm/)](https://docs.ragas.io/en/stable/howtos/applications/benchmark_llm/)

[[How to evaluate a Text to SQL Agent](https://docs.ragas.io/en/stable/howtos/applications/text2sql/)](https://docs.ragas.io/en/stable/howtos/applications/text2sql/)

[[How to Evaluate Your Prompt and Improve It](https://docs.ragas.io/en/stable/howtos/customizations/iterate_prompt/)](https://docs.ragas.io/en/stable/howtos/customizations/iterate_prompt/)

 [[How to Evaluate and Improve a RAG App](https://ragas--2293.org.readthedocs.build/en/2293/howtos/applications/evaluate-and-improve-rag/)](https://ragas--2293.org.readthedocs.build/en/2293/howtos/applications/evaluate-and-improve-rag/)

these reflect the new direction for Ragas and we would love all of your feedback here as well @petrov826  @Ice-YcY @KenanRustamov