Repository: explodinggradients/ragas
Issue: Problem with TestsetGenerator using Azure Open AI
Root cause: Validation Gap

Hello,
I have been trying to generate test questions but I always get this error: InvalidRequestError: Must provide an 'engine' or 'deployment_id' parameter to create a <class 'openai.api_resources.chat_completion.ChatCompletion'>
This is my code:
`azure_model = AzureChatOpenAI(
    deployment_name="ChatGPT16k",
    model_version="gpt-35-turbo-16k",
    openai_api_base="https://novaimsplayground.openai.azure.com/",
    openai_api_type="azure",
)
azure_embeddings = OpenAIEmbeddings(
    deployment="TextEmbeddings",
    model="text-embedding-ada-002",
    openai_api_base="https://novaimsplayground.openai.azure.com/",
    openai_api_type="azure",
)
critic_model = AzureChatOpenAI(
    deployment_name="ChatGPT",
    model_version="gpt-35-turbo",
    openai_api_base="https://novaimsplayground.openai.azure.com/",
    openai_api_type="azure",
)

testset_distribution = {
    "simple": 0.25,
    "reasoning": 0.5,
    "multi_context": 0.0,
    "conditional": 0.25,
}

chat_qa = 0.2

test_generator = TestsetGenerator(
    generator_llm=azure_model,
    critic_llm=critic_model,
    embeddings_model=azure_embeddings,
    testset_distribution=testset_distribution,
    chat_qa=chat_qa,
)
test_size = 10
testset = testsetgenerator.generate(documents, test_size=test_size)
`
All of these models were deployed and work correctly.
Can someone give me a hand?

Hey @miguel-cruz-27 , thanks for raising the issue. 
 
 I haven't tested it with Azure yet and it is working on some bugs [here](https://github.com/explodinggradients/ragas/pull/185). 

Can you confirm that you strictly followed the instructions mentioned [here](https://docs.ragas.io/en/latest/howtos/customisations/quickstart-azure-openai.html) including setting the environment variables?  

Would be super helpful if you can also paste and full code and error trace stack with the issue.   

Hi, 
Thanks for the comment. I am sorry, it was an issue in the code, instead of using test_generator I was using testsetgenerator that I must have defined previously. Still, now I am getting a new error:
`Traceback (most recent call last):

  File ~\anaconda3\envs\thesis\Lib\site-packages\IPython\core\interactiveshell.py:3526 in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  Cell In[42], line 38
    testset = test_generator.generate(documents, test_size=test_size)

  File ~\anaconda3\envs\thesis\Lib\site-packages\ragas\testset\testset_generator.py:305 in generate
    score = self._filter_context(text_chunk)

  File ~\anaconda3\envs\thesis\Lib\site-packages\ragas\testset\testset_generator.py:175 in _filter_context
    score = eval(results.generations[0][0].text.strip())

  File <string>:1
    Based on the provided context, I would assign a score of 2.
          ^
SyntaxError: invalid syntax
`
I have the environmental variables set:
`os.environ["OPENAI_API_TYPE"] = "azure"
os.environ["OPENAI_API_VERSION"] = "2023-07-01-preview"
os.environ["OPENAI_API_BASE"] = "https://novaimsplayground.openai.azure.com/"
os.environ["OPENAI_API_KEY"] = "..."`

The documents are defined like this:
`documents=[]
for i in range(df_base.shape[0]):
     documents.append(llama_index.Document(text=df_base["page_content"].iloc[i],metadata=ast.literal_eval(df_base["metadata"].iloc[i])))`, where df_base is a pandas dataframe that contains the text in a website ("page_content") and the link to that website ("metadata").
Even when I change the parameters of the distribution, I always get this error.
Could you give me a hand again?


I see, PR #185  here should fix this.  Once that's merged please update ragas using
`pip install -U git+https://github.com/explodinggradients/ragas.git`


Hi @miguel-cruz-27 We just made a new release, so you can just update ragas and it should solve the issue
`pip install -U ragas`

Thanks, that solved it!