Repository: explodinggradients/ragas
Issue: Bug in evaluate function 
Root cause: Validation Gap

This issue is related to #660 and we use the same model (VertexAI - gemini-pro) and same python. Ragas versión updated to 0.1.5.


**Describe the bug**
On the first issue we detected some weird behaviour when it comes to parallelization of different evaluation metrics. Now we detected a similar issue in the same metric. 

I have a dataset of 51 questions with ground truths. When I perform an evaluation on them I obtain a dataframe such as: 

|  question | ... |  answer_correctness |
|---|---|---|
| q1 | ... |  NaN |
| q2 | ... | NaN |
| q3 | ... | 0.15 |
| q4 | ... | 0.75 |
| ...| ... | ... |

But if I execute the first two questions i get: 
|  question | ... |  answer_correctness |
|---|---|---|
| q1 | ... | 0.15 |
| q2 | ... | 0.75 |

Meaning that scores are not assigned correctly at the end. 

While execution VertexAI raises two errors which corresponds to the two NaN values we observe in the first dataframe. Those errors are related to content filtering (We have some harmful questions in our dataset). 

Both the errors look like: 
````
File ".../venv/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py", line 822, in _send_message
    self._response_validator(
  File "[.../venv/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py", line 658](http://localhost:8888/lab/tree/venv/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py#line=657), in _validate_response
    raise ResponseValidationError(
vertexai.generative_models._generative_models.ResponseValidationError: The model response did not completed successfully.
Finish reason: 3.
Finish message: .
Safety ratings: [category: HARM_CATEGORY_HATE_SPEECH
probability: LOW
, category: HARM_CATEGORY_DANGEROUS_CONTENT
probability: NEGLIGIBLE
, category: HARM_CATEGORY_HARASSMENT
probability: HIGH
blocked: true
, category: HARM_CATEGORY_SEXUALLY_EXPLICIT
probability: NEGLIGIBLE
].
To protect the integrity of the chat session, the request and response were not added to chat history.
To skip the response validation, specify `model.start_chat(response_validation=False)`.
Note that letting blocked or otherwise incomplete responses into chat history might lead to future interactions being blocked by the service.
````

While this is specific to the VertexAI models, errors which are not controlled might behave the same way. 

We have a roundabout to this problem. Which is to redefine the VertexAI class such as: 

```python
from langchain_google_vertexai import VertexAI as VertexAI_


class BaseWrapper:

    @property
    def safety_settings(self):
        return {
            generative_models.HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: generative_models.HarmBlockThreshold.BLOCK_NONE,
            generative_models.HarmCategory.HARM_CATEGORY_HARASSMENT: generative_models.HarmBlockThreshold.BLOCK_NONE,
            generative_models.HarmCategory.HARM_CATEGORY_UNSPECIFIED: generative_models.HarmBlockThreshold.BLOCK_NONE,
            generative_models.HarmCategory.HARM_CATEGORY_HATE_SPEECH: generative_models.HarmBlockThreshold.BLOCK_NONE,
            generative_models.HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: generative_models.HarmBlockThreshold.BLOCK_NONE,
        }


class VertexAI(VertexAI_, BaseWrapper):
    """ This wrapper is used to ignore response block errors. They interfered with evaluation with ragas,
        as changed the executor order, making result order change and thus dataset not comparable at all.
        It also improves speed as no filtering is applied and ragas configurations interfered with errors.
        """
    def generate_prompt(self, *args, **kwargs):
        return super().generate_prompt(*args, **kwargs, safety_settings=self.safety_settings)
````

Once its redefined the filtering is no longer done when you call the method generate_prompt (Which is the one you use) and thus the metrics are assigned properly. 

I don't know how to fix the behaviour but I finally found what was happening. The stability analisys done in #660 is also affected by this behaviour and should be repeated to assert results. 

If you are able to control this behaviour with unexpected errors please indicate it on the release so we can filter harmfull content properly. 

Thank you again. 
 

Hey @abetatos thanks for reporting this error. Will look into it,
The coming versions of ragas will have the ability to improve metrics stability using self-consistency.