Repository: explodinggradients/ragas
Issue: AttributeError: 'AzureChatOpenAI' object has no attribute 'set_run_config'
Root cause: Interface Contract Mismatch

Hi, 

I've been encountering the error shown below when I try to run `evaluate()` inside one of my projects (run on a Docker env). For some reason it works when I try to run it on a Notebook (using the Azure example in the docs). I've made sure that the Azure Model on both the docker image and notebook have the same configuration.

```
  File "/usr/local/lib/python3.11/site-packages/ragas/evaluation.py", line 179, in evaluate
    [m.init(run_config) for m in metrics]
  File "/usr/local/lib/python3.11/site-packages/ragas/evaluation.py", line 179, in <listcomp>
    [m.init(run_config) for m in metrics]
     ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ragas/metrics/base.py", line 116, in init
    self.llm.set_run_config(run_config)
    ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'AzureChatOpenAI' object has no attribute 'set_run_config'
```

Implementation-wise, the notebook is purely straight-forward but for the one inside the docker, I call `evaluate()` inside an async function. I'm not sure if this would have an effect but I invoke `evaluate()` the same way as I did in the Notebook:

`ragas_result = evaluate(ragas_dataset, metrics=DEFAULT_METRICS, llm=DEFAULT_LLM_MODEL, embeddings=DEFAULT_EMBEDDINGS)`

Here's the config for the Azure model I'm using (I got the settings from the notebook and they match the one inside the docker image):

`AzureChatOpenAI(client=<openai.resources.chat.completions.Completions object at 0x157f7b910>, async_client=<openai.resources.chat.completions.AsyncCompletions object at 0x157f7ee10>, model_name='gpt-35-turbo', temperature=0.0, openai_api_key='...', openai_proxy='', request_timeout=60, azure_endpoint='https://...openai.azure.com/', deployment_name='...gpt-35-turbo', openai_api_version='2023-03-15-preview', openai_api_type='azure')
`

Here are also some library versions:
```
langchain==0.1.6
langchain-community==0.0.19
langchain-core==0.1.22
langchain-openai==0.0.5
ragas==0.1.0
python==3.11.5
```

Any help would be highly appreciated. I can provide more information if needed. Thank you.

Hey @sprt-kmelchor If I understand it correctly it's working for you w/o the docker but inside the containing it's throwing this error. Can you make sure that the ragas is of the same version inside the docker env too? ie 0.1

Hi @shahules786 yes that's correct. The ragas version in both instances (notebook and docker) are the same (v0.1.0).

is this issue resolved? I am using model defined in Llama index library, strangely am encountering this issue when i am executing in jupyter notebook, ragas version used is 0.1.0

> is this issue resolved? I am using model defined in Llama index library, strangely am encountering this issue when i am executing in jupyter notebook, ragas version used is 0.1.0

@sarathsurpur 
I was facing the same issue when using llama-index, but worked as soon as I used langchain described in the docs here: https://docs.ragas.io/en/stable/howtos/customisations/azure-openai.html

Cant say for sure whether its a llama-index support issue.

> > is this issue resolved? I am using model defined in Llama index library, strangely am encountering this issue when i am executing in jupyter notebook, ragas version used is 0.1.0
> 
> @sarathsurpur I was facing the same issue when using llama-index, but worked as soon as I used langchain described in the docs here: https://docs.ragas.io/en/stable/howtos/customisations/azure-openai.html
> 
> Cant say for sure whether its a llama-index support issue.

Issue is with openai version I suspect

thanks for raising this - seems like a code bug on our end, will get to it

Hi!

I have the same issue. Can you share the working version numbers?

Thanks

ok, solved from my side. I had a code issue. I received a code that had something like:

    # patch the models in all the metrics
    for m in metrics:
        m.__setattr__("llm", ragas_model)
        m.__setattr__("embeddings", embed_model)
        
removing that all worked as expected. 

I faced the same error on Notebook but when I upgraded my langchain, openai, azure-search-documents and ragas to the following:
`langchain==0.1.8
openai==1.12.0
azure-search-documents==11.6.0b1
pip install git+https://github.com/explodinggradients/ragas.git
`
it worked!


hey @sprt-kmelchor I'm not able to reproduce it but I think the issue - I think it could be a version problem as @bharathy-atlantic because we do wrap the LLM for you with `LangchainLLMWrapper` we detect that it's a langchain model or embedding (with `LangchainEmbeddingWrapper`).

if it's still an issue could you reply here?

thanks for sharing your fixes @pablocastilla. the latest version should do that setter for you so it is weird you had to manually do it - strange.

Hi @jjmachan, I just checked @pablocastilla's suggestion. Found the code snippet in our codebase and disabled that (I think this was how it was done in the older versions of Ragas when I checked some documentation). It ran albeit there are a few other errors but it is not relevant to this one anymore. I'll close this one now. Thank you all for the help and suggestions!

Hi, I got the same error `AttributeError: 'AzureChatOpenAI' object has no attribute 'set_run_config'` when run `evaluate`

Here are also some library versions:

```langchain== 0.1.8
langchain-community==0.0.28
langchain-core==0.1.31
langchain-openai==0.0.8
openai==1.12.0
ragas==0.1.4
```
And I also try @bharathy-atlantic 's suggestion, error still the same. Any help would be highly appreciated. Thank you.



Any news here?