Repository: explodinggradients/ragas
Issue: Azure Open AI Error when creating synthetic test data - Chat Completion Models Not Supported
Root cause: Interface Contract Mismatch

**Describe the bug**
I am trying to create a synthetic test data set using the Azure Open AI model endpoints.

I get an error message saying that the completion operation does not work with the specified models. I am using the GPT-4 models available on Azure which only work with the Chat Completion API and not the Completion API.

Is there a way to leverage the Azure Open AI models which use the Chat Completion API?

Ragas version: ragas==0.0.22
Python version: Python 3.10.4

**Code to Reproduce**
```python
from ragas.testset import TestsetGenerator
from langchain.embeddings import OpenAIEmbeddings
from langchain.chat_models import ChatOpenAI
from ragas.llms import LangchainLLM
# Import Azure OpenAI
from langchain_openai import AzureOpenAI
# documents = load your documents

# Add custom llms and embeddings

# Create an instance of Azure OpenAI
# Replace the deployment name with your own

generator_llm = LangchainLLM(llm=AzureOpenAI(deployment_name="gpt-35-turbo",
    model_name="gpt-3.5-turbo",
    api_version="2023-12-01-preview"))
critic_llm = LangchainLLM(llm=AzureOpenAI(deployment_name="gpt-4-1106",
    model_name="gpt-4-1106",
    api_version="2023-12-01-preview"))
embeddings_model = OpenAIEmbeddings(
            api_key="",
            api_base="",
            api_type="azure",
            api_version="2023-12-01-preview")

# Change resulting question type distribution
testset_distribution = {
    "simple": 0.25,
    "reasoning": 0.5,
    "multi_context": 0.0,
    "conditional": 0.25,
}

# percentage of conversational question
chat_qa = 0.2


test_generator = TestsetGenerator(
    generator_llm=generator_llm,
    critic_llm=critic_llm,
    embeddings_model=embeddings_model,
    testset_distribution=testset_distribution,
    chat_qa=chat_qa,
)

testset = test_generator.generate(documents, test_size=5)

```
**Error trace**
```
---------------------------------------------------------------------------
BadRequestError                           Traceback (most recent call last)
File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/ragas/async_utils.py:42, in run_async_tasks(tasks, show_progress, progress_bar_desc)
     41 try:
---> 42     outputs: List[Any] = loop.run_until_complete(_gather())
     43 except Exception as e:
     44     # run the operation w/o tqdm on hitting a fatal
     45     # may occur in some environments where tqdm.asyncio
     46     # is not supported

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/nest_asyncio.py:99, in _patch_loop.<locals>.run_until_complete(self, future)
     97     raise RuntimeError(
     98         'Event loop stopped before Future completed.')
---> 99 return f.result()

File ~/.pyenv/versions/3.10.4/lib/python3.10/asyncio/futures.py:201, in Future.result(self)
    200 if self._exception is not None:
--> 201     raise self._exception
    202 return self._result

File ~/.pyenv/versions/3.10.4/lib/python3.10/asyncio/tasks.py:234, in Task.__step(***failed resolving arguments***)
    233     else:
--> 234         result = coro.throw(exc)
    235 except StopIteration as exc:

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/ragas/async_utils.py:39, in run_async_tasks.<locals>._gather()
     38 async def _gather() -> List[Any]:
---> 39     return await asyncio.gather(*tasks_to_execute)

File ~/.pyenv/versions/3.10.4/lib/python3.10/asyncio/tasks.py:304, in Task.__wakeup(self, future)
    303 try:
--> 304     future.result()
    305 except BaseException as exc:
    306     # This may also be a cancellation.

File ~/.pyenv/versions/3.10.4/lib/python3.10/asyncio/tasks.py:232, in Task.__step(***failed resolving arguments***)
    229 if exc is None:
    230     # We use the `send` method directly, because coroutines
    231     # don't have `__iter__` and `__next__` methods.
--> 232     result = coro.send(None)
    233 else:

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/ragas/llms/langchain.py:132, in LangchainLLM.generate_completions(self, prompts, callbacks)
    131     ps = [p.format() for p in prompts]
--> 132     result = await self.llm.agenerate(ps, callbacks=callbacks)
    133 else:  # if BaseChatModel

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/langchain_core/language_models/llms.py:893, in BaseLLM.agenerate(self, prompts, stop, callbacks, tags, metadata, run_name, **kwargs)
    892 run_managers = [r[0] for r in run_managers]
--> 893 output = await self._agenerate_helper(
    894     prompts, stop, run_managers, bool(new_arg_supported), **kwargs
    895 )
    896 return output

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/langchain_core/language_models/llms.py:757, in BaseLLM._agenerate_helper(self, prompts, stop, run_managers, new_arg_supported, **kwargs)
    751     await asyncio.gather(
    752         *[
    753             run_manager.on_llm_error(e, response=LLMResult(generations=[]))
    754             for run_manager in run_managers
    755         ]
    756     )
--> 757     raise e
    758 flattened_outputs = output.flatten()

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/langchain_core/language_models/llms.py:741, in BaseLLM._agenerate_helper(self, prompts, stop, run_managers, new_arg_supported, **kwargs)
    739 try:
    740     output = (
--> 741         await self._agenerate(
    742             prompts,
    743             stop=stop,
    744             run_manager=run_managers[0] if run_managers else None,
    745             **kwargs,
    746         )
    747         if new_arg_supported
    748         else await self._agenerate(prompts, stop=stop)
    749     )
    750 except BaseException as e:

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/langchain_openai/llms/base.py:401, in BaseOpenAI._agenerate(self, prompts, stop, run_manager, **kwargs)
    400 else:
--> 401     response = await self.async_client.create(prompt=_prompts, **params)
    402     if not isinstance(response, dict):

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/openai/resources/completions.py:1133, in AsyncCompletions.create(self, model, prompt, best_of, echo, frequency_penalty, logit_bias, logprobs, max_tokens, n, presence_penalty, seed, stop, stream, suffix, temperature, top_p, user, extra_headers, extra_query, extra_body, timeout)
   1091 @required_args(["model", "prompt"], ["model", "prompt", "stream"])
   1092 async def create(
   1093     self,
   (...)
   1131     timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN,
   1132 ) -> Completion | AsyncStream[Completion]:
-> 1133     return await self._post(
   1134         "/completions",
   1135         body=maybe_transform(
   1136             {
   1137                 "model": model,
   1138                 "prompt": prompt,
   1139                 "best_of": best_of,
   1140                 "echo": echo,
   1141                 "frequency_penalty": frequency_penalty,
   1142                 "logit_bias": logit_bias,
   1143                 "logprobs": logprobs,
   1144                 "max_tokens": max_tokens,
   1145                 "n": n,
   1146                 "presence_penalty": presence_penalty,
   1147                 "seed": seed,
   1148                 "stop": stop,
   1149                 "stream": stream,
   1150                 "suffix": suffix,
   1151                 "temperature": temperature,
   1152                 "top_p": top_p,
   1153                 "user": user,
   1154             },
   1155             completion_create_params.CompletionCreateParams,
   1156         ),
   1157         options=make_request_options(
   1158             extra_headers=extra_headers, extra_query=extra_query, extra_body=extra_body, timeout=timeout
   1159         ),
   1160         cast_to=Completion,
   1161         stream=stream or False,
   1162         stream_cls=AsyncStream[Completion],
   1163     )

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/openai/_base_client.py:1536, in AsyncAPIClient.post(self, path, cast_to, body, files, options, stream, stream_cls)
   1533 opts = FinalRequestOptions.construct(
   1534     method="post", url=path, json_data=body, files=await async_to_httpx_files(files), **options
   1535 )
-> 1536 return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/openai/_base_client.py:1315, in AsyncAPIClient.request(self, cast_to, options, stream, stream_cls, remaining_retries)
   1306 async def request(
   1307     self,
   1308     cast_to: Type[ResponseT],
   (...)
   1313     remaining_retries: Optional[int] = None,
   1314 ) -> ResponseT | _AsyncStreamT:
-> 1315     return await self._request(
   1316         cast_to=cast_to,
   1317         options=options,
   1318         stream=stream,
   1319         stream_cls=stream_cls,
   1320         remaining_retries=remaining_retries,
   1321     )

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/openai/_base_client.py:1392, in AsyncAPIClient._request(self, cast_to, options, stream, stream_cls, remaining_retries)
   1390         await err.response.aread()
-> 1392     raise self._make_status_error_from_response(err.response) from None
   1394 return self._process_response(
   1395     cast_to=cast_to,
   1396     options=options,
   (...)
   1399     stream_cls=stream_cls,
   1400 )

BadRequestError: Error code: 400 - {'error': {'code': 'OperationNotSupported', 'message': 'The completion operation does not work with the specified model, gpt-4. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.'}}

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
Cell In[17], line 51
     40 chat_qa = 0.2
     43 test_generator = TestsetGenerator(
     44     generator_llm=generator_llm,
     45     critic_llm=critic_llm,
   (...)
     48     chat_qa=chat_qa,
     49 )
---> 51 testset = test_generator.generate(documents, test_size=5)

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/ragas/testset/testset_generator.py:351, in TestsetGenerator.generate(self, documents, test_size)
    344 nodes = (
    345     self._get_neighbour_node(curr_node, neighbor_nodes)
    346     if size > 1 and evolve_type != "multi_context"
    347     else [curr_node]
    348 )
    350 text_chunk = " ".join([node.get_content() for node in nodes])
--> 351 score = self._filter_context(text_chunk)
    352 if not score:
    353     continue

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/ragas/testset/testset_generator.py:192, in TestsetGenerator._filter_context(self, context)
    190 human_prompt = SCORE_CONTEXT.format(context=context)
    191 prompt = ChatPromptTemplate.from_messages([human_prompt])
--> 192 results = self.critic_llm.generate(prompts=[prompt])
    193 output = results.generations[0][0].text.strip()
    194 score = load_as_score(output)

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/ragas/llms/langchain.py:211, in LangchainLLM.generate(self, prompts, n, temperature, callbacks)
    209     return self._generate_multiple_completions(prompts, n, callbacks)
    210 else:  # call generate_completions n times to mimic multiple completions
--> 211     list_llmresults = run_async_tasks(
    212         [self.generate_completions(prompts, callbacks) for _ in range(n)]
    213     )
    215     # fill results as if the LLM supported multiple completions
    216     generations = []

File ~/.pyenv/versions/3.10.4/envs/euca-api/lib/python3.10/site-packages/ragas/async_utils.py:47, in run_async_tasks(tasks, show_progress, progress_bar_desc)
     42     outputs: List[Any] = loop.run_until_complete(_gather())
     43 except Exception as e:
     44     # run the operation w/o tqdm on hitting a fatal
     45     # may occur in some environments where tqdm.asyncio
     46     # is not supported
---> 47     raise RuntimeError("Fatal error occurred while running async tasks.", e)
     48 return outputs

RuntimeError: ('Fatal error occurred while running async tasks.', BadRequestError("Error code: 400 - {'error': {'code': 'OperationNotSupported', 'message': 'The completion operation does not work with the specified model, gpt-4. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.'}}"))
```


**Expected behavior**
The code should create the synthetic data from the data provided.

**Additional context**
Add any other context about the problem here.


Same issue here. Did you manage to solve this?

Edit:
It's because the Azure Interface is wrong. 
Instead of 
`from langchain_openai import AzureOpenAI`
use
`from langchain.chat_models import AzureChatOpenAI`
and use that for the critic_llm

@uwaisiqbal Same issue here. Any workaround? @cosmin-novac is not relevant to me.