Repository: explodinggradients/ragas
Issue: [R-305] Incorrect API key provided Azure OpenAI for TestsetGenerator
Root cause: Configuration Error

I am trying to use Azure Openai to create a test dataset. But for some reasons the `generate_with_langchain_docs` give the following error:

`unable to apply transformation: Error code: 401 - {'error': {'message': 'Incorrect API key provided: 0ba2ec70********************407d. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}
`
when I test my api key using `azure_llm.invoke("tell me a jok about weather")` it work.

here is my code
  
```
import os
from ragas.testset import TestsetGenerator
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper

from langchain_openai.chat_models import AzureChatOpenAI
from langchain_openai.embeddings import AzureOpenAIEmbeddings
from langchain_community.document_loaders import WebBaseLoader


config = {
    "base_url": "https://xxxxx.openai.azure.com",
    "model_deployment": "my-gpt-4o-rag",
    "model_name": "gpt-4o",
    "embedding_deployment": "my-text-embedding-ada-002",
    "embedding_name": "text-embedding-ada-002",  }

loader = WebBaseLoader("https://www.advancinganalytics.co.uk/")
documents = loader.load()
for document in documents:
    document.metadata['filename'] = document.metadata['source']

azure_llm = AzureChatOpenAI(
    openai_api_version= "2024-08-01-preview",
    azure_endpoint=config["base_url"],
    azure_deployment=config["model_deployment"],
    model=config["model_name"],
    validate_base_url=False,
)

response = azure_llm.invoke("tell me a jok about the weather") #---> this works

azure_llm = LangchainLLMWrapper(azure_llm)
generator = TestsetGenerator.from_langchain(azure_llm)
testset = generator.generate_with_langchain_docs(
    documents,
    testset_size=5,
)

```

`Ragas=0.2.1` and `langchain_openai==0.2.2`


<sub>[R-305](https://linear.app/exploding-gradients/issue/R-305/incorrect-api-key-provided-azure-openai-for-testsetgenerator)</sub>

<!-- Answer -->
The error you're encountering suggests that the API key being used by the `generate_with_langchain_docs` method is incorrect or not being passed correctly. Here are a few things you can check and try:

1. **API Key Configuration**: Ensure that the API key is correctly set in your environment or configuration. The `LangchainLLMWrapper` and `AzureChatOpenAI` should be using the same API key. Double-check that the API key is correctly configured in your environment variables or wherever it is being set.

2. **Environment Variables**: If you're using environment variables to store your API key, make sure they are correctly set and accessible in the context where `generate_with_langchain_docs` is being called.

3. **Wrapper Initialization**: When you wrap `AzureChatOpenAI` with `LangchainLLMWrapper`, ensure that the wrapper is correctly passing the API key to the underlying model. There might be a discrepancy in how the wrapper handles the API key compared to direct invocation.

4. **Version Compatibility**: Although there are no known issues between `ragas==0.2.1` and `langchain_openai==0.2.2`, ensure that all dependencies are up to date and compatible. Sometimes, updating to the latest versions can resolve unexpected issues.

5. **Debugging**: Add debugging logs to check if the API key is being altered or not passed correctly when using `generate_with_langchain_docs`. This can help identify where the issue might be occurring.

If these steps do not resolve the issue, consider isolating the problem by testing the API key with a minimal setup to ensure it works as expected in different parts of your code.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I've already check the api key by calling the `AzureChatOpenAI.invoke` method and it works just fine. 

@llvll0hsen I have exactly the same issue! It seems that the problem lies in the `llm_factory`. It falls back on `ChatOpenAI`, instead of `AzureChatOpenAI`.  The docs here https://docs.ragas.io/en/stable/howtos/customizations/customize_models/#customize-models give an overview, but it still fails. @jjmachan can we expect a hotfix?

Same here. The LLM on Azure works by itself, but not when it is embedded in TestsetGenerator, instead of using Azure API key, it uses OpenAI API key.

hey let me look into this at this ASAP and I will get back

> hey let me look into this at this ASAP and I will get back

Thanks! BTW, it is the same for embedding_factory. It would be nice to pass the llm and embedding model through the TestsetGenerator parameter and actually **use them** in the default transforms.

I was having the same error. It seems like the llm_factory defaults to OpenAI and there's no support in that function for AzureOpenAI. I altered the code locally to try to see if it would accept an AzureOpenAI client but no luck. In the previous version (0.1.21) you used to be able to pass your LLM and embedding client directly to TestsetGenerator, but that seems to have changed in the latest version. 


Are there any updates on this?

I figured out the problem. I implemented a temporary fix locally just to have the ability to produce a test set, but it is not a long-term solution. I am willing to spend some time working on a fix if nobody else is. 


hey @hunter-walden2113 really curious about what the fix was - I'll also take this up this week

@jjmachan : the problem originates here: 

`ragas.testset.synthesizers.generate.TestsetGenerator.generate_with_langchain_docs`. 

The current code defaults to using `default_transforms()` if the user does not pass an argument for the `transforms` parameter. 
When `default_transforms()` is called, it instantiates all the required `Extractor` objects in `transforms.__init__.py`, where the default llm in the `Extractor` class is instantiated by calling `llm_factory`. Since the user is not using the dafault llm (OpenAI), and therefore not setting the required environmental variables, they are getting the "API Key not found error". 

I fixed this by allowing the user to pass their models into TestsetGenerator, and providing those model clients to the Extractors, instead of using the default llms. 

To fix this I did the following:

1. Added `embedding_model` as an attribute to `TestsetGenerator`.
2. Added `embedding_model: LangchainEmbeddings` as a parameter to `TestsetGenerator.from_langchain`
3. Changed the return from `TestsetGenerator.from_langchain` to `return cls(LangchainLLMWrapper(llm), LangchainEmbeddingsWrapper(embedding_model), knowledge_graph)`
4.  Added both an `llm` and `embedding_model` parameter to `TestsetGenerator.generate_with_langchain_docs`
5. Added `llm` and `embedding_model` as parameters to `testset.transforms.__init__.default_transforms`
6. Passed the llm and embedding arguments to each of the Extractors in default_transforms as appropriate. 

I will schedule some time to talk with you to see what you think of the fix and how to improve it. 

closing this since the fixes have been merged 🙂 
thanks again @hunter-walden2113 ❤️