Repository: explodinggradients/ragas
Issue: Error while implementing AzureChatOpenAI() using the documented code provided. Missing all values of Answer relevancy and some values of other metrics
Root cause: External Dependency Breakage

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**

Exception raised in Job[49]: TimeoutError() this is one sort of error I am getting 
Exception raised in Job[29]: BadRequestError(Unsupported data type) this is the other 
 The exceptions are around 20 in total each of this format

All the values of answer of Answer relevancy are missing. Some values from the rest metrics are missing 
Ragas version:0.1.19
Python version:3.11
OpenAI Version:1.33.0

**Code to Reproduce**
from langchain_openai.chat_models import AzureChatOpenAI
from langchain_openai.embeddings import AzureOpenAIEmbeddings
from ragas import evaluate
azure_model = AzureChatOpenAI(
    openai_api_version="2024-02-15-preview",
    azure_endpoint=base_url,
    azure_deployment="",
    model="gpt-4o",
    validate_base_url=False,
    api_key="",
)
azure_embeddings = AzureOpenAIEmbeddings(
    openai_api_version="2024-02-15-preview",
    azure_endpoint=base_url,
    azure_deployment="",
    model="textembeddingada002",
    api_key="",
)
from ragas.metrics import (
    context_precision,
    answer_relevancy,
    faithfulness,
    context_recall,
)
metrics = [ 
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
]
from datasets import load_dataset

amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")
print(amnesty_qa)

from datasets import load_dataset
from ragas import RunConfig


run_config = RunConfig(timeout=300, log_tenacity=True)
amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")
amnesty_qa
result = evaluate(
    amnesty_qa["eval"], metrics=metrics, llm=azure_model, embeddings=azure_embeddings,run_config=run_config,
)
**Error trace**

**Expected behavior**
I am expecting to run fully as this is the code provided by the team to test, I want to test with new data but with the provide sample code ragas isn't working for me


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Greeting -->
Hey @koushik9121! I'm here to help you with any bugs, questions, or contributions. Let's work together to solve the issue you're facing.

<!-- Answer -->
To resolve the `TimeoutError` and `BadRequestError (Unsupported data type)` exceptions when implementing `AzureChatOpenAI` and evaluating metrics like answer relevancy, you can follow these steps:

1. **Ensure Correct Configuration**: Double-check that your Azure OpenAI configurations are correct, including the `base_url`, `model_deployment`, `model_name`, `embedding_deployment`, and `embedding_name`.

2. **Handle TimeoutError**: Increase the timeout settings for the API calls if possible. This can be done by setting a higher timeout value in the `AzureChatOpenAI` and `AzureOpenAIEmbeddings` initialization.

3. **Handle BadRequestError (Unsupported data type)**: Ensure that the data types being passed to the API are supported. This might involve converting data to the correct format before making API calls.

Here is the updated code with these considerations:

```python
from langchain_openai.chat_models import AzureChatOpenAI
from langchain_openai.embeddings import AzureOpenAIEmbeddings
from ragas import evaluate
import os

# Ensure your Azure OpenAI key is set in the environment
os.environ["AZURE_OPENAI_API_KEY"] = "your-azure-openai-api-key"

# Azure OpenAI configurations
azure_configs = {
    "base_url": "https://<your-endpoint>.openai.azure.com/",
    "model_deployment": "your-deployment-name",
    "model_name": "your-model-name",
    "embedding_deployment": "your-deployment-name",
    "embedding_name": "text-embedding-ada-002",  # most likely
}

# Initialize the chat model with a higher timeout
azure_model = AzureChatOpenAI(
    openai_api_version="2023-05-15",
    azure_endpoint=azure_configs["base_url"],
    azure_deployment=azure_configs["model_deployment"],
    model=azure_configs["model_name"],
    validate_base_url=False,
    request_timeout=60  # Increase the timeout to 60 seconds
)

# Initialize the embeddings with a higher timeout
azure_embeddings = AzureOpenAIEmbeddings(
    openai_api_version="2023-05-15",
    azure_endpoint=azure_configs["base_url"],
    azure_deployment=azure_configs["embedding_deployment"],
    model=azure_configs["embedding_name"],
    request_timeout=60  # Increase the timeout to 60 seconds
)

# Load the dataset
from datasets import load_dataset
amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")

# Import metrics
from ragas.metrics import (
    context_precision,
    answer_relevancy,
    faithfulness,
    context_recall,
)
from ragas.metrics.critique import harmfulness

# List of metrics to use
metrics = [
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
    harmfulness,
]

# Evaluate the dataset
result = evaluate(
    amnesty_qa["eval"], metrics=metrics, llm=azure_model, embeddings=azure_embeddings
)

# Convert results to pandas DataFrame
df = result.to_pandas()
df.head()
```

By increasing the `request_timeout` parameter, you can mitigate `TimeoutError` issues. Ensure that the data passed to the API is in the correct format to avoid `BadRequestError (Unsupported data type)` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/customisations/azure-openai.ipynb)</sup>.

Additionally, there are known issues or limitations with the `ragas` library version 0.1.19 that could affect the evaluation of metrics like answer relevancy. Specifically, the `_calculate_score` method has a potential issue where if all generated questions are empty, it logs a warning and sets the score to `NaN`. Similarly, the `_ascore` method has a check for `None` answers, which also results in a `NaN` score <sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_relevance.py)</sup>.

For additional logging and debugging, you can enable debug mode by setting the environment variable `RAGAS_DEBUG` to `true` and use the `patch_logger` function to enable debug logging for specific modules <sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/utils.py)</sup><sup>[[4]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/_analytics.py)</sup><sup>[[5]](https://github.com/explodinggradients/ragas/blob/main/src/experimental/ragas_experimental/testset/loaders/test_sync.py)</sup>:

```sh
export RAGAS_DEBUG=true
```

```python
from ragas.utils import patch_logger
import logging

patch_logger('your_module_name', logging.DEBUG)
```

This setup will help you capture detailed information about the execution and diagnose issues with missing metric values.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I have checked by the keys and the names by making a test run using azure_model.invoke() and the api was returning correct answer. But what you need to understand is i am trying to run the code provided by the RAGAS team itself and still errors are being there. Help me with this issue 

<!-- Answer -->
To resolve the `TimeoutError` and `BadRequestError (Unsupported data type)` exceptions when running the provided RAGAS team code using `AzureChatOpenAI` and evaluating metrics like answer relevancy, ensure the following:

1. **TimeoutError**: Increase the timeout settings for the Azure OpenAI API calls. This can be done by setting the `timeout` parameter in the `AzureChatOpenAI` and `AzureOpenAIEmbeddings` instances.

2. **BadRequestError (Unsupported data type)**: Ensure that the data types being passed to the API are supported. This typically involves ensuring that the inputs are in the correct format (e.g., strings for text inputs).

Here is the updated code with the necessary changes:

```python
from langchain_openai.chat_models import AzureChatOpenAI
from langchain_openai.embeddings import AzureOpenAIEmbeddings
from ragas import evaluate

# Configuration for Azure OpenAI
azure_configs = {
    "base_url": "https://<your-endpoint>.openai.azure.com/",
    "model_deployment": "your-deployment-name",
    "model_name": "your-model-name",
    "embedding_deployment": "your-deployment-name",
    "embedding_name": "text-embedding-ada-002",  # most likely
}

# Initialize the chat model with a timeout setting
azure_model = AzureChatOpenAI(
    openai_api_version="2023-05-15",
    azure_endpoint=azure_configs["base_url"],
    azure_deployment=azure_configs["model_deployment"],
    model=azure_configs["model_name"],
    validate_base_url=False,
    timeout=60  # Set a timeout of 60 seconds
)

# Initialize the embeddings with a timeout setting
azure_embeddings = AzureOpenAIEmbeddings(
    openai_api_version="2023-05-15",
    azure_endpoint=azure_configs["base_url"],
    azure_deployment=azure_configs["embedding_deployment"],
    model=azure_configs["embedding_name"],
    timeout=60  # Set a timeout of 60 seconds
)

# Load the dataset
from datasets import load_dataset
amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")

# Import metrics
from ragas.metrics import (
    context_precision,
    answer_relevancy,
    faithfulness,
    context_recall,
)
from ragas.metrics.critique import harmfulness

# List of metrics to use
metrics = [
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
    harmfulness,
]

# Initialize the metrics with LLMs and Embeddings
from ragas.run_config import RunConfig
from ragas.metrics.base import MetricWithLLM, MetricWithEmbeddings

def init_ragas_metrics(metrics, llm, embedding):
    for metric in metrics:
        if isinstance(metric, MetricWithLLM):
            metric.llm = llm
        if isinstance(metric, MetricWithEmbeddings):
            metric.embeddings = embedding
        run_config = RunConfig()
        metric.init(run_config)

# Wrappers for LLM and Embeddings
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper

init_ragas_metrics(
    metrics,
    llm=LangchainLLMWrapper(azure_model),
    embedding=LangchainEmbeddingsWrapper(azure_embeddings),
)

# Evaluate the dataset
result = evaluate(
    amnesty_qa["eval"], metrics=metrics, llm=azure_model, embeddings=azure_embeddings
)

# Convert results to pandas DataFrame
df = result.to_pandas()
df.head()
```

Ensure that the `azure_configs` dictionary is correctly populated with your Azure OpenAI endpoint details. The `timeout` parameter is set to 60 seconds, but you can adjust this value based on your needs. Additionally, the `init_ragas_metrics` function ensures that the metrics are properly initialized with the LLM and embeddings <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/customisations/azure-openai.ipynb)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Serializing the data might help