Repository: explodinggradients/ragas
Issue: issue with metrics evaluation  in case of any exception
Root cause: Algorithmic Error

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Evaluating dataset with multiple metrics using evaluate function. for now calling evaluate function for each metrics separately[because of issue -> [878](https://github.com/explodinggradients/ragas/issues/878) ]
Observation - 

- If there is no exception(eg. context length exceeds) than result looks fine.
- if there is at-lease one exception, that result seems completely different and not stable. (using evaluate with raise_exception = False), I think the orders messed up.


Ragas version: 0.1.7
Python version: 3.10.9

**Code to Reproduce**

```python
import ragas
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
    context_relevancy,
    answer_correctness
)

from datasets import Dataset 
from ragas import evaluate
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

ragas_metrics = [ faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
    context_relevancy,
    answer_correctness]

def ragas_evaluate(df_res, metrics, query_col = "Question", truth_ans_col = "Answer" ,
                       llm = ChatOpenAI(model="gpt-3.5-turbo"),
                       embed_model = OpenAIEmbeddings(model="text-embedding-3-small"),
                       raise_exceptions = True,
                       timeout=10, max_retries=3, max_wait=10,
                       is_async = True,
                        callbacks = None
                   ):

        data_samples = {
            'question': df_res[query_col].to_list(),
            'answer': df_res.rag_answer.to_list(),
            'contexts' : df_res.context.to_list(),
            'ground_truth': df_res[truth_ans_col].to_list()
        }

        run_config = ragas.RunConfig(timeout=timeout, max_retries=max_retries, max_wait=max_wait)

        dataset = Dataset.from_dict(data_samples)
   
        for idx, metric in enumerate(metrics):
            print(f">> {idx+1} : {metric.name}")
            score = evaluate(dataset, metrics= [metric], llm = llm,
                        embeddings= embed_model, raise_exceptions=raise_exceptions, run_config=run_config,
                             is_async=is_async, callbacks=callbacks)
            time.sleep(1)
            if idx == 0:
                df_score = score.to_pandas()
            else:
                df_score = df_score.merge(score.to_pandas()[['question',metric.name]], on = 'question', how = "left")

        return df_score


# for now showing with only faithfullness and answer_relevancy
## without any data point which may cause exception
df_eval = ragas_evaluate(df_response.iloc[list(range(20))], ragas_metrics[0:2], raise_exceptions=False)
df_eval

## with a data point which cause exception - openai.BadRequestError: Error code: 400 [context_length_exceeded]
df_eval_ex = ragas_evaluate(df_res.iloc[list(range(20))+ [68]], ragas_metrics[0:2], raise_exceptions=False)
df_eval_ex
``` 
<br/><br/>
### Result of df_eval 

<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>question</th>
      <th>answer</th>
      <th>contexts</th>
      <th>ground_truth</th>
      <th>faithfulness</th>
      <th>answer_relevancy</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>How has Intel's total net sales fluctuated ove...</td>
      <td>Intel's total net sales decreased from approxi...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Intel's total net sales have fluctuated as fol...</td>
      <td>1.000000</td>
      <td>0.910671</td>
    </tr>
    <tr>
      <th>1</th>
      <td>What are the significant factors that have led...</td>
      <td>The significant factors that have led to chang...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>In the latest 10-Q for the quarter ended Septe...</td>
      <td>1.000000</td>
      <td>0.994862</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Has there been a substantial shift in Intel's ...</td>
      <td>Yes, there has been a substantial shift in Int...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Yes, there has been a substantial shift in Int...</td>
      <td>0.500000</td>
      <td>0.891755</td>
    </tr>
    <tr>
      <th>3</th>
      <td>How has Intel's revenue from chip sales seen v...</td>
      <td>Intel's revenue from chip sales has seen signi...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Intel's revenue from chip sales has seen the f...</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Are there identifiable trends in Intel's Data ...</td>
      <td>There is a downward trend in Intel's Data Cent...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Based on the provided documents, there is a tr...</td>
      <td>0.750000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>How does Intel's research and development spen...</td>
      <td>Intel's research and development spending in t...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>In the most recent quarter, which ended on Sep...</td>
      <td>0.428571</td>
      <td>0.861540</td>
    </tr>
    <tr>
      <th>6</th>
      <td>What legal proceedings or potential liabilitie...</td>
      <td>Intel's 10-Q reports have disclosed several le...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Intel's 10-Q filings disclose several legal pr...</td>
      <td>1.000000</td>
      <td>0.965009</td>
    </tr>
    <tr>
      <th>7</th>
      <td>Are there any significant alterations in Intel...</td>
      <td>Yes, there was a significant increase in Intel...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Yes, there have been alterations in Intel's de...</td>
      <td>0.750000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>In Amazon's Q2 2023 10-Q, how does the discuss...</td>
      <td>The discussion on employee compensation and be...</td>
      <td>[Sub question: How does Amazon discuss employe...</td>
      <td>The discussion on employee compensation and be...</td>
      <td>0.875000</td>
      <td>0.846506</td>
    </tr>
    <tr>
      <th>9</th>
      <td>Has Intel reported any significant business ac...</td>
      <td>Yes, Intel reported the acquisition of Tower S...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Yes, Intel reported significant business acqui...</td>
      <td>1.000000</td>
      <td>0.899457</td>
    </tr>
    <tr>
      <th>10</th>
      <td>How have inventory levels at Intel changed ove...</td>
      <td>Intel's inventory levels have decreased from $...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Over the quarters provided in the context, Int...</td>
      <td>1.000000</td>
      <td>0.893291</td>
    </tr>
    <tr>
      <th>11</th>
      <td>Outline the risk factors for Intel's business,...</td>
      <td>The risk factors for Intel's business include ...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>The risk factors for Intel's business as outli...</td>
      <td>1.000000</td>
      <td>0.840134</td>
    </tr>
    <tr>
      <th>12</th>
      <td>How has Amazon's total net sales changed over ...</td>
      <td>IDK</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Amazon's total net sales have changed over tim...</td>
      <td>NaN</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>13</th>
      <td>What are the major factors contributing to the...</td>
      <td>The major factors contributing to the change i...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>The major factors contributing to the change i...</td>
      <td>1.000000</td>
      <td>0.970940</td>
    </tr>
    <tr>
      <th>14</th>
      <td>Has there been any significant change in Amazo...</td>
      <td>Yes, there has been a significant change in Am...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Yes, there has been a significant change in Am...</td>
      <td>0.800000</td>
      <td>0.844631</td>
    </tr>
    <tr>
      <th>15</th>
      <td>How has Amazon's revenue from online retail se...</td>
      <td>IDK</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Amazon's revenue from online retail services, ...</td>
      <td>NaN</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>16</th>
      <td>Can any trends be identified in Amazon's AWS (...</td>
      <td>Yes, there is a positive revenue trend in Amaz...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Yes, trends can be identified in Amazon's AWS ...</td>
      <td>1.000000</td>
      <td>0.893554</td>
    </tr>
    <tr>
      <th>17</th>
      <td>What is the impact of foreign exchange rates o...</td>
      <td>- For Q1 2023, foreign exchange rates had a mi...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>The impact of foreign exchange rates on Amazon...</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>18</th>
      <td>Are there any notable changes in Amazon's liqu...</td>
      <td>Yes, there are notable changes in Amazon's liq...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Yes, there are notable changes in Amazon's liq...</td>
      <td>1.000000</td>
      <td>0.919368</td>
    </tr>
    <tr>
      <th>19</th>
      <td>How does Amazon's R&amp;D expenditure in the most ...</td>
      <td>IDK</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>In the most recent quarter ended September 30,...</td>
      <td>NaN</td>
      <td>0.854005</td>
    </tr>
  </tbody>
</table>
</div>

* Expected result. index 12, 15, 19 have "IDK" in response and accordingly faithfulness and answer_relevancy  is coming NaN/0.

<br/><br/>
### Result of df_eval_ex


<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>question</th>
      <th>answer</th>
      <th>contexts</th>
      <th>ground_truth</th>
      <th>faithfulness</th>
      <th>answer_relevancy</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>How has Intel's total net sales fluctuated ove...</td>
      <td>Intel's total net sales decreased from approxi...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Intel's total net sales have fluctuated as fol...</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>1</th>
      <td>What are the significant factors that have led...</td>
      <td>The significant factors that have led to chang...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>In the latest 10-Q for the quarter ended Septe...</td>
      <td>1.000000</td>
      <td>0.910671</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Has there been a substantial shift in Intel's ...</td>
      <td>Yes, there has been a substantial shift in Int...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Yes, there has been a substantial shift in Int...</td>
      <td>1.000000</td>
      <td>0.986719</td>
    </tr>
    <tr>
      <th>3</th>
      <td>How has Intel's revenue from chip sales seen v...</td>
      <td>Intel's revenue from chip sales has seen signi...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Intel's revenue from chip sales has seen the f...</td>
      <td>0.000000</td>
      <td>0.891747</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Are there identifiable trends in Intel's Data ...</td>
      <td>There is a downward trend in Intel's Data Cent...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Based on the provided documents, there is a tr...</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>How does Intel's research and development spen...</td>
      <td>Intel's research and development spending in t...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>In the most recent quarter, which ended on Sep...</td>
      <td>0.750000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>What legal proceedings or potential liabilitie...</td>
      <td>Intel's 10-Q reports have disclosed several le...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Intel's 10-Q filings disclose several legal pr...</td>
      <td>0.750000</td>
      <td>0.861468</td>
    </tr>
    <tr>
      <th>7</th>
      <td>Are there any significant alterations in Intel...</td>
      <td>Yes, there was a significant increase in Intel...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Yes, there have been alterations in Intel's de...</td>
      <td>1.000000</td>
      <td>0.965009</td>
    </tr>
    <tr>
      <th>8</th>
      <td>In Amazon's Q2 2023 10-Q, how does the discuss...</td>
      <td>The discussion on employee compensation and be...</td>
      <td>[Sub question: How does Amazon discuss employe...</td>
      <td>The discussion on employee compensation and be...</td>
      <td>0.750000</td>
      <td>0.770066</td>
    </tr>
    <tr>
      <th>9</th>
      <td>Has Intel reported any significant business ac...</td>
      <td>Yes, Intel reported the acquisition of Tower S...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Yes, Intel reported significant business acqui...</td>
      <td>0.857143</td>
      <td>0.846523</td>
    </tr>
    <tr>
      <th>10</th>
      <td>How have inventory levels at Intel changed ove...</td>
      <td>Intel's inventory levels have decreased from $...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Over the quarters provided in the context, Int...</td>
      <td>1.000000</td>
      <td>0.899457</td>
    </tr>
    <tr>
      <th>11</th>
      <td>Outline the risk factors for Intel's business,...</td>
      <td>The risk factors for Intel's business include ...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>The risk factors for Intel's business as outli...</td>
      <td>1.000000</td>
      <td>0.891580</td>
    </tr>
    <tr>
      <th>12</th>
      <td>How has Amazon's total net sales changed over ...</td>
      <td>IDK</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Amazon's total net sales have changed over tim...</td>
      <td>1.000000</td>
      <td>0.840090</td>
    </tr>
    <tr>
      <th>13</th>
      <td>What are the major factors contributing to the...</td>
      <td>The major factors contributing to the change i...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>The major factors contributing to the change i...</td>
      <td>NaN</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>14</th>
      <td>Has there been any significant change in Amazo...</td>
      <td>Yes, there has been a significant change in Am...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Yes, there has been a significant change in Am...</td>
      <td>0.687500</td>
      <td>0.961019</td>
    </tr>
    <tr>
      <th>15</th>
      <td>How has Amazon's revenue from online retail se...</td>
      <td>IDK</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Amazon's revenue from online retail services, ...</td>
      <td>0.800000</td>
      <td>0.900011</td>
    </tr>
    <tr>
      <th>16</th>
      <td>Can any trends be identified in Amazon's AWS (...</td>
      <td>Yes, there is a positive revenue trend in Amaz...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Yes, trends can be identified in Amazon's AWS ...</td>
      <td>NaN</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>17</th>
      <td>What is the impact of foreign exchange rates o...</td>
      <td>- For Q1 2023, foreign exchange rates had a mi...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>The impact of foreign exchange rates on Amazon...</td>
      <td>1.000000</td>
      <td>0.893554</td>
    </tr>
    <tr>
      <th>18</th>
      <td>Are there any notable changes in Amazon's liqu...</td>
      <td>Yes, there are notable changes in Amazon's liq...</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>Yes, there are notable changes in Amazon's liq...</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>19</th>
      <td>How does Amazon's R&amp;D expenditure in the most ...</td>
      <td>IDK</td>
      <td>[Sub question: By first identifying and quotin...</td>
      <td>In the most recent quarter ended September 30,...</td>
      <td>0.857143</td>
      <td>0.852599</td>
    </tr>
    <tr>
      <th>20</th>
      <td>From the most recent 10-Q, how do Intel's comm...</td>
      <td>IDK</td>
      <td>[Sub question: What are Intel's commitments in...</td>
      <td>The most recent 10-Q does not provide specific...</td>
      <td>NaN</td>
      <td>0.000000</td>
    </tr>
  </tbody>
</table>
</div>

* index 20 data point is causing exception and result messed up.
    - both metrics coming NaN for data index 0. which is wrong.
    - data index 3 -> faithfulness is 0 and answer_relevancy is non zero.
    - Data index - 12, 15, 19 -> rag answer is 'IDK' but still both metrics is coming high non zero.

**Error trace**

**Expected behavior**
* Result should be nearly same for df_eval and df_eval_ex (except data point 20). 
* I think ordering is getting messed up while collecting the result of different data points.
* tried without using evaluate function means calling each metrics **score** function and handling exception explicitly , result are near fine but performance is not. 


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Hello, I encountered the same problem recently due to exception on content filtering with openai.

While investigating I found the origin of the problem here:
https://github.com/explodinggradients/ragas/blob/2d793651f778b6c0da07a834e9ce2765be13cc9f/src/ragas/executor.py#L77

In `Runner`, jobs are normally wrapped with a counter to be able to reorder the metrics at the end because asyncio does not respect the order: 
https://github.com/explodinggradients/ragas/blob/2d793651f778b6c0da07a834e9ce2765be13cc9f/src/ragas/executor.py#L110-L114
https://github.com/explodinggradients/ragas/blob/2d793651f778b6c0da07a834e9ce2765be13cc9f/src/ragas/executor.py#L144-L145

But in the event of an exception and when `raise_exceptions=False`, the counter is not used: the index -1 is returned, which by default places the NaN at the beginning, and shakes up the entire list of metrics.
https://github.com/explodinggradients/ragas/blob/2d793651f778b6c0da07a834e9ce2765be13cc9f/src/ragas/executor.py#L77-L89


A possible solution would be to move the `np.nan` and `try except` into the function‎ `Executor.wrap_callable_with_index` instead of
‎`Runner._aresults` to allow returning the counter even in the event of an exception.

@baptiste-pasquier Thanks for RCA. Have you tried the workaround possible solution/code? 

Here is the fix code: https://github.com/baptiste-pasquier/ragas/commit/167b9c5d86db69940ff34eeeb8b1107be162e9b4

# Code for testing

```python
import ragas
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
)

data = {
    "question": [
        "What is the capital of France?",
        "What is the capital of France?",
        "What is the capital of France?",
        "What is the capital of France?",
    ],
    "answer": [
        "Paris is the capital of France.",
        "",
        "London is the capital of France.",
        "x" * 99999,  # Context length error
    ],
    "contexts": [
        ["Paris is the capital of France."],
        ["Paris is the capital of France."],
        ["Paris is the capital of France."],
        ["Paris is the capital of France."],
    ],
    "ground_truth": [
        "Paris",
        "Paris",
        "Paris",
        "Paris",
    ],
}
dataset = Dataset.from_dict(data)

metrics = [faithfulness, answer_relevancy]
run_config = ragas.RunConfig(timeout=10, max_retries=1, max_wait=10)
```

# Test 1 (without exception)
```python
result = evaluate(
    dataset.select(range(3)),  # Run without the last row
    metrics=metrics,
    llm=llm,
    embeddings=embeddings,
    raise_exceptions=False,
    run_config=run_config,
)
result.to_pandas()
```

## Result before and after fix:

question | answer | contexts | ground_truth | faithfulness | answer_relevancy
-- | -- | -- | -- | -- | --
What is the capital of France? | Paris is the capital of France. | [Paris is the capital of France.] | Paris | 1.0 | 1.0
What is the capital of France? |   | [Paris is the capital of France.] | Paris | NaN | 1.0
What is the capital of France? | London is the capital of France. | [Paris is the capital of France.] | Paris | 0.0 | 0.0

# Test 2 (with exception)
A context length exception is created at the last row of the dataset.

```python
result = evaluate(
    dataset,  # Run on the entire dataset
    metrics=metrics,
    llm=llm,
    embeddings=embeddings,
    raise_exceptions=False,
    run_config=run_config,
)
result.to_pandas()
```
## Result before fix:

question | answer | contexts | ground_truth | faithfulness | answer_relevancy
-- | -- | -- | -- | -- | --
What is the capital of France? | Paris is the capital of France. | [Paris is the capital of France.] | Paris | $\color{red}\textsf{NaN}$ | $\color{red}\textsf{NaN}$
What is the capital of France? |   | [Paris is the capital of France.] | Paris | 1.0 | 1.0
What is the capital of France? | London is the capital of France. | [Paris is the capital of France.] | Paris | NaN | 1.0
What is the capital of France? | xxxxxxxxxxxxxx... | [Paris is the capital of France.] | Paris | 0.0 | 0.0

_The NaNs appear in the first row, which is unexpected behavior (the other metrics are shifted in the table)._

## Result after fix:

question | answer | contexts | ground_truth | faithfulness | answer_relevancy
-- | -- | -- | -- | -- | --
What is the capital of France? | Paris is the capital of France. | [Paris is the capital of France.] | Paris | 1.0 | 1.0
What is the capital of France? |   | [Paris is the capital of France.] | Paris | NaN | 1.0
What is the capital of France? | London is the capital of France. | [Paris is the capital of France.] | Paris | 0.0 | 0.0
What is the capital of France? | xxxxxxxxxxxxxx... | [Paris is the capital of France.] | Paris | $\color{red}\textsf{NaN}$ | $\color{red}\textsf{NaN}$

_The NaNs appear in the last row, which is the expected behavior._






Yes, i have tried the similar fix, it seems working.