Repository: explodinggradients/ragas
Issue: [0.2.0-beta] issue with faithfulness metric - Exception raised in Job[1]: AttributeError('dict' object has no attribute 'verdict')
Root cause: Interface Contract Mismatch

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
A clear and concise description of what the bug is.
Came across error when running very basic evaluation on faithfulness metric.

Ragas version: 0.2.0-beta
Python version: 3.10.13

**Code to Reproduce**
Share code to reproduce the issue
```
import os

# Set the environment variable
os.environ['RAGAS_DO_NOT_TRACK'] = 'true'

# You can now access the environment variable like this:
do_not_track = os.environ.get('RAGAS_DO_NOT_TRACK', 'false')
print(f"RAGAS_DO_NOT_TRACK is set to: {do_not_track}")

from datasets import Dataset, load_dataset
import os
from ragas import evaluate
from ragas.metrics import faithfulness, answer_correctness
import json

from langchain_community.chat_models import BedrockChat
from langchain_community.embeddings import BedrockEmbeddings
import boto3
from ragas import evaluate
import nest_asyncio  # CHECK NOTES

from ragas.metrics import (
    faithfulness,
)
from ragas.run_config import RunConfig
from botocore.config import Config

import pandas as pd
import ragas
import numpy as np

# Create a RunConfig with a specified timeout
bedrock_config = Config(read_timeout=1000)

config = {
    "credentials_profile_name": "your-profile-name",  # E.g "default"
    "region_name": "us-west-2",  # E.g. "us-east-1"
    "model_id":"anthropic.claude-3-sonnet-20240229-v1:0",
    "model_kwargs": {"temperature": 0.4},
}

def get_bedrock_client(region):
    session = boto3.Session(profile_name='personal')
    bedrock_client = session.client("bedrock-runtime", region_name=region, config=bedrock_config)
    return bedrock_client

bedrock_client = get_bedrock_client('us-west-2')

bedrock_model = BedrockChat(
    region_name=config["region_name"],
    # endpoint_url=f"https://bedrock-runtime.{config['region_name']}.amazonaws.com",
    client=bedrock_client,
    model_id=config["model_id"],
    model_kwargs=config["model_kwargs"],
)

# init the embeddings
bedrock_embeddings = BedrockEmbeddings(
    # credentials_profile_name=config["credentials_profile_name"],
    client=bedrock_client,
    region_name=config["region_name"],
)

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
    'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}

dataset = Dataset.from_dict(data_samples)
print(dataset)

# faithfulness
score = evaluate(dataset,metrics=[faithfulness],llm=bedrock_model,embeddings=bedrock_embeddings)
```
**Error trace**
```
Exception raised in Job[1]: AttributeError('dict' object has no attribute 'verdict')
Exception raised in Job[0]: AttributeError('dict' object has no attribute 'verdict')
```
**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Answer -->
A similar issue was previously reported and closed. The issue titled ['list' object has no attribute 'get' in _context_precision.py](https://github.com/explodinggradients/ragas/issues/533) was resolved by a pull request that handled non-dict outputs <sup>[[1]](https://github.com/explodinggradients/ragas/issues/533)</sup>.

In your case, the error `AttributeError('dict' object has no attribute 'verdict')` suggests that the code is attempting to access the `verdict` attribute on a dictionary object, which is not valid. This might be due to the structure of the data being processed by the `faithfulness` metric.

To resolve this, ensure that the data being passed to the `faithfulness` metric is structured correctly. The `verdict` attribute should be part of a class or object, not a dictionary. You might need to check how the `StatementFaithfulnessAnswer` class is being used and ensure that the data conforms to the expected structure, where `verdict` is an attribute of an object, not a dictionary key.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I'm passing the exact data sample in the quickstart section. It should not be a problem of input.

should we close this now @linzzzzzz ?

Yes. I can confirm it's working as expect now.