Repository: explodinggradients/ragas
Issue: Azure OpenAI NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}
Root cause: Configuration Error

#### Description of bug:
Hi all, I am trying to run the evaluation of my datasets using the following four metrics: Faithfulness, Answer Relevance, Answer Semantic Similarity, and Answer Correctness. I have configured the necessary settings to set up the Azure OpenAI models to run the evaluation, as shown below. However, when it comes to evaluating the "Answer Correctness" metric, it seems like the backend code can't find the resource needed to run it. Does anyone know what is happening, or did I configure anything wrongly? Thank you so much!
 
Ragas version: 0.0.22
Python version: 3.10.6

#### Code to Reproduce:
```
from langchain.chat_models import AzureChatOpenAI
from langchain.embeddings import AzureOpenAIEmbeddings
from ragas.llms import LangchainLLM
from ragas import evaluate

from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    AnswerSimilarity,
    AnswerCorrectness,
)

answer_similarity = AnswerSimilarity()
answer_correctness = AnswerCorrectness()

# List of metrics we're going to use
metrics = [
    faithfulness,
    answer_relevancy,
    answer_similarity,
    answer_correctness,
]

azure_model = AzureChatOpenAI(
    deployment_name="xxx",
    model="gpt-35-turbo",
    openai_api_base="https://xxx.openai.azure.com/",
    openai_api_type="azure",
)

# Wrapper around azure_model
ragas_azure_model = LangchainLLM(azure_model)

# Only for answer_relevancy
azure_embeddings = AzureOpenAIEmbeddings(
    deployment="xxx_embed",
    model="text-embedding-ada-002",
    openai_api_base="https://xxx.openai.azure.com/",
    openai_api_type="azure",
)

# Embeddings can be used as they are
answer_relevancy.embeddings = azure_embeddings
answer_similarity.embeddings = azure_embeddings
answer_correctness.embeddings = azure_embeddings

for m in metrics:
    m.__setattr__("llm", ragas_azure_model)

# Create an empty dictionary to store results for each split
evaluation_results = {}

# Iterate over each split in your DatasetDict
for split_name, dataset in my_dataset_dict.items():
.........................
    # Iterate over each sample in the dataset
    for sample in dataset:
        try:
            # Convert the sample dictionary back to a dataset-like object
            sample_dataset = Dataset.from_dict({key: [value] for key, value in sample.items()})

            # Perform evaluation on the current sample
            result = evaluate(
                sample_dataset,
                metrics=metrics,
            )

          .........................................

        except RequestException as e:
            print(f"Error occurred while evaluating sample {sample}: {e}")
            continue
.....................
```

#### Error trace:
```
evaluating with [answer_correctness]

  0%|          | 0/1 [00:03<?, ?it/s]

---------------------------------------------------------------------------

NotFoundError                             Traceback (most recent call last)

Cell In[28], line 24

     22 sample_dataset = Dataset.from_dict({key: [value] for key, value in sample.items()})

     23 # Perform evaluation on the current sample

---> 24 result = evaluate(

     25     sample_dataset,

     26     metrics=metrics,

     27 )

     29 # Update the sums of scores and the count of samples

     30 for metric in sum_scores.keys():

File /opt/conda/lib/python3.10/site-packages/ragas/evaluation.py:101, in evaluate(dataset, metrics, column_map)

     99         binary_metrics.append(metric.name)

    100     print(f"evaluating with [{metric.name}]")

--> 101     scores.append(metric.score(dataset).select_columns(metric.name))

    103 # log the evaluation event

    104 metrics_names = [m.name for m in metrics]

File /opt/conda/lib/python3.10/site-packages/ragas/metrics/base.py:76, in Metric.score(self, dataset, callbacks)

     74 with trace_as_chain_group(f"ragas_{self.name}", callback_manager=cm) as group:

     75     for batch in tqdm(self.get_batches(len(dataset))):

---> 76         score = self._score_batch(dataset.select(batch), callbacks=group)

     77         scores.extend(score)

     79 return dataset.add_column(f"{self.name}", scores)

File /opt/conda/lib/python3.10/site-packages/ragas/metrics/_answer_correctness.py:139, in AnswerCorrectness._score_batch(self, dataset, callbacks, callback_group_name)

    135         score = np.nan

    137     f1_score.append(score)

--> 139 similarity_scores = self.answer_similarity._score_batch(dataset)  # type: ignore

    140 scores_stacked = np.vstack([f1_score, similarity_scores])

    141 scores = np.average(

    142     scores_stacked,

    143     axis=0,

    144     weights=self.weights,

    145 )
 
File /opt/conda/lib/python3.10/site-packages/ragas/metrics/_answer_similarity.py:81, in AnswerSimilarity._score_batch(self, dataset, callbacks, callback_group_name)

     79     scores = np.array(self.embeddings.predict(inputs))

     80 else:

---> 81     embeddings_1 = np.array(self.embeddings.embed_documents(ground_truths))

     82     embeddings_2 = np.array(self.embeddings.embed_documents(answers))

     83     similarity = embeddings_1 @ embeddings_2.T

File /opt/conda/lib/python3.10/site-packages/langchain_community/embeddings/openai.py:668, in OpenAIEmbeddings.embed_documents(self, texts, chunk_size)

    665 # NOTE: to keep things simple, we assume the list may contain texts longer

    666 #       than the maximum context and use length-safe embedding function.

    667 engine = cast(str, self.deployment)

--> 668 return self._get_len_safe_embeddings(texts, engine=engine)

File /opt/conda/lib/python3.10/site-packages/langchain_community/embeddings/openai.py:494, in OpenAIEmbeddings._get_len_safe_embeddings(self, texts, engine, chunk_size)

    492 batched_embeddings: List[List[float]] = []

    493 for i in _iter:

--> 494     response = embed_with_retry(

    495         self,

    496         input=tokens[i : i + _chunk_size],

    497         **self._invocation_params,

    498     )

    499     if not isinstance(response, dict):

    500         response = response.dict() 

File /opt/conda/lib/python3.10/site-packages/langchain_community/embeddings/openai.py:116, in embed_with_retry(embeddings, **kwargs)

    114 """Use tenacity to retry the embedding call."""

    115 if is_openai_v1():

--> 116     return embeddings.client.create(**kwargs)

    117 retry_decorator = _create_retry_decorator(embeddings)

    119 @retry_decorator

    120 def _embed_with_retry(**kwargs: Any) -> Any:

File /opt/conda/lib/python3.10/site-packages/openai/resources/embeddings.py:108, in Embeddings.create(self, input, model, encoding_format, user, extra_headers, extra_query, extra_body, timeout)

    102         embedding.embedding = np.frombuffer(  # type: ignore[no-untyped-call]

    103             base64.b64decode(data), dtype="float32"

    104         ).tolist()

    106     return obj

--> 108 return self._post(

    109     "/embeddings",

    110     body=maybe_transform(params, embedding_create_params.EmbeddingCreateParams),

    111     options=make_request_options(

    112         extra_headers=extra_headers,

    113         extra_query=extra_query,

    114         extra_body=extra_body,

    115         timeout=timeout,

    116         post_parser=parser,

    117     ),

    118     cast_to=CreateEmbeddingResponse,

    119 )

File /opt/conda/lib/python3.10/site-packages/openai/_base_client.py:1167, in SyncAPIClient.post(self, path, cast_to, body, options, files, stream, stream_cls)

   1153 def post(

   1154     self,

   1155     path: str,

   (...)

   1162     stream_cls: type[_StreamT] | None = None,

   1163 ) -> ResponseT | _StreamT:

   1164     opts = FinalRequestOptions.construct(

   1165         method="post", url=path, json_data=body, files=to_httpx_files(files), **options

   1166     )

-> 1167     return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))

File /opt/conda/lib/python3.10/site-packages/openai/_base_client.py:856, in SyncAPIClient.request(self, cast_to, options, remaining_retries, stream, stream_cls)

    847 def request(

    848     self,

    849     cast_to: Type[ResponseT],

   (...)

    854     stream_cls: type[_StreamT] | None = None,

    855 ) -> ResponseT | _StreamT:

--> 856     return self._request(

    857         cast_to=cast_to,

    858         options=options,

    859         stream=stream,

    860         stream_cls=stream_cls,

    861         remaining_retries=remaining_retries,

    862     )

File /opt/conda/lib/python3.10/site-packages/openai/_base_client.py:947, in SyncAPIClient._request(self, cast_to, options, remaining_retries, stream, stream_cls)

    944         err.response.read()

    946     log.debug("Re-raising status error")

--> 947     raise self._make_status_error_from_response(err.response) from None

    949 return self._process_response(

    950     cast_to=cast_to,

    951     options=options,

   (...)

    954     stream_cls=stream_cls,

    955 )
 
NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}
```


cc @shahules786 @jjmachan , really appreciate if anyone can offer any form of help!

I am having the same error when trying to pass the llm and embeddings model using Azure OpenAI configuration to the evaluate function. This happens to all metrics.

@danieljohnxon , Can you provide me the sample on how do you create a your own dataset?. I am getting issue when try to use my own datasetdict
my_dataset_dict

same here, when I try to run answer_correctness with azure open ai model getting this error, would really appreciate if any suggestions here to tackle this

hey @danieljohnxon @edgarlcs and @Manikanta5112 really sorry about this, but could you guys try out the latest v0.1 version. That has the fix for this.

here are the docs https://docs.ragas.io/en/latest/howtos/customisations/azure-openai.html
and here is how you can install from source
`pip install git+https://github.com/explodinggradients/ragas.git`

hoping that fixed this, feel free to reopen if not :)

hi @jjmachan I'm getting this same error with ragas 0.1.11, but only for answer_relevancy. The same code works if I change to other metrics (faithfulness, answer_correctness, answer_similarity) or if I downgrade to version 0.1.10.

@mariamedp are you are using azure openai? does this only happen with Azure for you?

@jjmachan I have this only on azure openai

Having the same issue on Azure OpenAI. Error code is unclear:
```
Exception raised in Job[0]: NotFoundError(Error code: 404 - {'statusCode': 404, 'message': 'Resource not found'})
```