Repository: explodinggradients/ragas
Issue: Failed to parse output. Returning None.
Root cause: Interface Contract Mismatch

[ X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When calling evaluate on Ragas using AWS Bedrock, there are some examples that give the warning: "Failed to parse output. Returning None." and error: "__root__ -> 1 -> reason
  field required (type=value_error.missing)" for a few examples. 

Ragas version: 0.1.13
Python version: 3.10.14

**Code to Reproduce**
from langchain_community.chat_models import BedrockChat
from langchain_community.embeddings import BedrockEmbeddings
from datasets import load_dataset

amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")

config = {
    "region_name": "us-east-1",  # E.g. "us-east-1"
    "model_id": 'anthropic.claude-3-haiku-20240307-v1:0',  # E.g "anthropic.claude-v2"
    "model_kwargs": {"temperature": 0.4},
}

bedrock_model = BedrockChat(
    region_name=config["region_name"],
    endpoint_url=f"https://bedrock-runtime.{config['region_name']}.amazonaws.com",
    model_id=config["model_id"],
    model_kwargs=config["model_kwargs"],
)


bedrock_embeddings = BedrockEmbeddings(
    region_name=config["region_name"],
)

from ragas import evaluate
import nest_asyncio  # CHECK NOTES

nest_asyncio.apply()

result = evaluate(
    amnesty_qa["eval"],
    metrics=metrics,
    llm=bedrock_model,
    embeddings=bedrock_embeddings,
)

result

**Error trace**
Failed to parse output. Returning None.
Failed to parse output. Returning None.
Exception raised in Job[34]: ValidationError(2 validation errors for ContextPrecisionVerifications
__root__ -> 1 -> reason
  field required (type=value_error.missing)
__root__ -> 1 -> verdict
  field required (type=value_error.missing))
Failed to parse output. Returning None.
Failed to parse output. Returning None.
Failed to parse output. Returning None.
Failed to parse output. Returning None.
Failed to parse output. Returning None.
Failed to parse output. Returning None.
Failed to parse output. Returning None.
Failed to parse output. Returning None.

The above is for a subset of 20 examples that are evaluated. 

**Expected behavior**
None of the warnings above should come


**Additional context**
This error/warning doesn't stop execution and a lot of examples do get evaluated smoothly. It is just that some examples can't be evaluated and results in a NaN value for the metric. 

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Greeting -->
Hello, @surengunturuamazon! I'm here to help you with any bugs, questions, or contributions.

<!-- Answer -->
To resolve the "Failed to parse output. Returning None." and "__root__ -> 1 -> reason field required (type=value_error.missing)" errors when calling `evaluate` on Ragas using AWS Bedrock, you need to ensure that the JSON output conforms to the expected schema and includes all required fields. The error message indicates that the "reason" field is missing in the JSON output.

Here is a step-by-step approach to resolve the issue:

1. **Check the Schema**: Ensure that the schema used in `get_json_format_instructions` includes the "reason" field as a required field.

2. **Modify the Schema**: If the "reason" field is not included, add it to the schema.

3. **Update the Prompt**: Ensure that the prompt used to generate the output includes instructions to include the "reason" field.

Here is an example of how you can modify the schema and prompt:

```python
def get_json_format_instructions(pydantic_object: t.Type[TBaseModel]) -> str:
    # Copy schema to avoid altering original Pydantic schema.
    schema = {k: v for k, v in pydantic_object.schema().items()}

    # Remove extraneous fields.
    reduced_schema = schema
    if "title" in reduced_schema:
        del reduced_schema["title"]
    # Ensure json in context is well-formed with double quotes.
    schema_str = json.dumps(reduced_schema)

    # Add "reason" field to the schema if it's missing
    if "properties" in reduced_schema and "reason" not in reduced_schema["properties"]:
        reduced_schema["properties"]["reason"] = {"type": "string"}
        if "required" in reduced_schema:
            reduced_schema["required"].append("reason")
        else:
            reduced_schema["required"] = ["reason"]

    resp = JSON_FORMAT_INSTRUCTIONS.format(schema=schema_str)
    return resp

# Ensure the prompt includes instructions to include the "reason" field
FIX_OUTPUT_FORMAT = Prompt(
    name="",
    instruction="Below, the Completion did not satisfy the constraints given in the Prompt.",
    output_format_instruction="Ensure the output includes a 'reason' field.",
    input_keys=["prompt", "completion"],
    output_key="fixed_completion",
)
```

Additionally, when using AWS Bedrock with Ragas, ensure that you have correctly configured the `BedrockChat` and `BedrockEmbeddings` classes from Langchain, as shown in the following example:

```python
from langchain_community.chat_models import BedrockChat
from langchain_community.embeddings import BedrockEmbeddings

config = {
    "region_name": "us-east-1",  # E.g. "us-east-1"
    "model_id": "anthropic.claude-3-haiku-20240307-v1:0",  # E.g "anthropic.claude-v2"
    "model_kwargs": {"temperature": 0.4},
}

bedrock_model = BedrockChat(
    region_name=config["region_name"],
    endpoint_url=f"https://bedrock-runtime.{config['region_name']}.amazonaws.com",
    model_id=config["model_id"],
    model_kwargs=config["model_kwargs"],
)
# init the embeddings
bedrock_embeddings = BedrockEmbeddings(
    region_name=config["region_name"],
)
```

Furthermore, ensure that the `EvaluatorChain` class in Ragas is correctly validating the required fields. The `_validate` method in the `EvaluatorChain` class should check for the presence of all necessary fields, including "reason":

```python
def _validate(
        self,
        input: dict[str, t.Any],
        question_key: str = "question",
        prediction_key: str = "answer",
        context_key: str = "contexts",
    ) -> None:
        # validate each example
        required_columns = EVALMODE_TO_COLUMNS[self.metric.evaluation_mode]
        if "question" in required_columns and question_key not in input:
            raise ValueError(
                f'"{question_key}" is required in each example'
                f"for the metric[{self.metric.name}] you have chosen."
            )
        if "answer" in required_columns and prediction_key not in input:
            raise ValueError(
                f'"{prediction_key}" is required in each prediction'
                f"for the metric[{self.metric.name}] you have chosen."
            )
        if "contexts" in required_columns and context_key not in input:
            raise ValueError(
                f'"{context_key}" is required in each prediction for the '
                f"metric[{self.metric.name}] you have chosen."
            )
        if "ground_truth" in required_columns and "ground_truth" not in input:
            raise ValueError(
                f'"ground_truth" is required in each prediction for the '
                f"metric[{self.metric.name}] you have chosen."
            )
        if "reason" in required_columns and "reason" not in input:
            raise ValueError(
                f'"reason" is required in each prediction for the '
                f"metric[{self.metric.name}] you have chosen."
            )
```

By ensuring that the schema includes the "reason" field, updating the prompt to instruct the model to include this field, and correctly configuring the Bedrock classes and validation methods, you should be able to resolve the errors <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/output_parser.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/customisations/aws-bedrock.ipynb)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/integrations/langchain.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Closing this as resolved in more recent versions with timeouts.