Repository: explodinggradients/ragas
Issue: Open AI error with version 0.1.0
Root cause: External Dependency Breakage

I am trying to get the following metrics using ragas 0.1.0 , open AI version as 1.12.0, llama index as 0.8.51.post1
answer_relevancy, answer_correctness but getting this error 
**openai.NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}**
       

my code is something like this
from ragas.metrics import (
    context_precision,
    answer_relevancy,
    faithfulness,
    context_recall,
    answer_correctness,
    context_relevancy,
    answer_correctness
)

from ragas import evaluate

from langchain.chat_models import AzureChatOpenAI
from langchain.embeddings import OpenAIEmbeddings

azure_model = AzureChatOpenAI(
        deployment_name="gpt-model",
        model="gpt-35-turbo",
        openai_api_base="xxxxx",
        openai_api_type="azure",
        openai_api_key="xxxxxxx",
        openai_api_version='xxxxx',
    )

azure_embeddings = OpenAIEmbeddings(
        deployment="text-embedding-ada-002",
        model="text-embedding-ada-002",
        openai_api_base="xxxxx",
        openai_api_type="azure",
        openai_api_key="xxxxx",
        openai_api_version='xxxxx',
    )

final_df = df_eval[['question','ground_truths','answer','contexts']]
# Huggingface dataset
dataset = Dataset.from_pandas(final_df)
metrics = [
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall,
        context_relevancy,
        #answer_correctness
    ]

result = evaluate(dataset, metrics=metrics, embeddings = azure_embeddings, llm = azure_model)


The above code is a sample code, may be I would have missed something to add but it is working for context_precision, faithfulness, context_recall.

Please let me know what latest versions shall I use for ragas, OpenAI


I am getting the same error with similar code :  my keys works fine on normal convIR method
openai.NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}

Hey @jeny1292 @KINJALMARU16 can you both install ragas from the source and try again if it works?

> Hey @jeny1292 @KINJALMARU16 can you both install ragas from the source and try again if it works?

Hello, are you asking me to download the latest version? I mean the git install?


@jeny1292 `pip install git+https://github.com/explodinggradients/ragas.git`

> @jeny1292 `pip install git+https://github.com/explodinggradients/ragas.git`

I am still getting the same error even after installing from the source.

Hey @KINJALMARU16 @jeny1292 Can you both ensure that you're following the guide [here](https://docs.ragas.io/en/stable/howtos/customisations/azure-openai.html) and reinstall ragas from source again (if you did the prev install week ago or so). 
Asking because it's working for me. 

If you have langchain 0.1 installed, try this:

# Load your environment variables
azure_api_key = os.environ.get('AZURE_OPENAI_API_KEY')
api_type = os.environ.get('OPENAI_API_TYPE')
azure_openai_api_version = os.environ.get('OPENAI_API_VERSION')
azure_api_endpoint = os.environ.get('AZURE_OPENAI_ENDPOINT')
azure_gpt3_model = os.environ.get('GPT3_MODEL_NAME')
azure_gpt4_model = os.environ.get('GPT4_MODEL_NAME')
azure_embed_model = os.environ.get('EMBED_MODEL_NAME')

os.environ['AZURE_OPENAI_API_KEY'] = azure_api_key
os.environ['AZURE_OPENAI_ENDPOINT'] = azure_api_endpoint
os.environ["OPENAI_API_VERSION"] = azure_openai_api_version

from langchain_openai.embeddings import AzureOpenAIEmbeddings
from langchain_openai.chat_models import AzureChatOpenAI
from datasets import Dataset

from ragas import evaluate
from ragas.metrics import (
    context_precision,
    answer_relevancy,
    faithfulness,
    context_recall,
    #answer_correctness,
    context_relevancy,
    #answer_correctness
)

# If you are using Azure OpenAI Chat
azure_model = AzureChatOpenAI(
    api_key=azure_api_key,
    azure_endpoint=azure_api_endpoint,
    azure_deployment=azure_gpt3_model, # gpt-35-turbo Make sure this to use your deployment name for the model
    api_version=azure_openai_api_version, # typically 2023-07-01-preview
)

# If you are using Azure OpenAI Embeddings
azure_embeddings = AzureOpenAIEmbeddings(
    api_key=azure_api_key,
    azure_endpoint=azure_api_endpoint,
    azure_deployment=azure_embed_model, # text-embeddings-ada-002 Make sure this to use your deployment name for the model
    api_version=azure_openai_api_version, # typically 2023-07-01-preview
)

# The latest version of RAGAs is looking from ground_truth rather than ground_truths
# and it should just be a string rather than a sequence of strings
final_df = df_eval[['question','ground_truths','answer','contexts']]

dataset = Dataset.from_pandas(final_df)

metrics = [
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
    context_relevancy,
    #answer_correctness
]

result = evaluate(
    dataset=dataset,
    column_map={"question": "question", "contexts": "contexts", "answer": "answer", "ground_truth": "ground_truths"},
    llm=azure_model,
    embeddings=azure_embeddings,
    metrics=metrics,
    is_async=True,
)

final_results = result.to_pandas()

final_results

> Hey @KINJALMARU16 @jeny1292 Can you both ensure that you're following the guide [here](https://docs.ragas.io/en/stable/howtos/customisations/azure-openai.html) and reinstall ragas from source again (if you did the prev install week ago or so). Asking because it's working for me.

Thank you, it is doing the calculation now and completes it...but it also shows this error
**RuntimeError: Event loop is closed
Invalid JSON response. Expected dictionary with key 'Attributed'**

> If you have langchain 0.1 installed, try this:
> 
> # Load your environment variables
> azure_api_key = os.environ.get('AZURE_OPENAI_API_KEY') api_type = os.environ.get('OPENAI_API_TYPE') azure_openai_api_version = os.environ.get('OPENAI_API_VERSION') azure_api_endpoint = os.environ.get('AZURE_OPENAI_ENDPOINT') azure_gpt3_model = os.environ.get('GPT3_MODEL_NAME') azure_gpt4_model = os.environ.get('GPT4_MODEL_NAME') azure_embed_model = os.environ.get('EMBED_MODEL_NAME')
> 
> os.environ['AZURE_OPENAI_API_KEY'] = azure_api_key os.environ['AZURE_OPENAI_ENDPOINT'] = azure_api_endpoint os.environ["OPENAI_API_VERSION"] = azure_openai_api_version
> 
> from langchain_openai.embeddings import AzureOpenAIEmbeddings from langchain_openai.chat_models import AzureChatOpenAI from datasets import Dataset
> 
> from ragas import evaluate from ragas.metrics import ( context_precision, answer_relevancy, faithfulness, context_recall, #answer_correctness, context_relevancy, #answer_correctness )
> 
> # If you are using Azure OpenAI Chat
> azure_model = AzureChatOpenAI( api_key=azure_api_key, azure_endpoint=azure_api_endpoint, azure_deployment=azure_gpt3_model, # gpt-35-turbo Make sure this to use your deployment name for the model api_version=azure_openai_api_version, # typically 2023-07-01-preview )
> 
> # If you are using Azure OpenAI Embeddings
> azure_embeddings = AzureOpenAIEmbeddings( api_key=azure_api_key, azure_endpoint=azure_api_endpoint, azure_deployment=azure_embed_model, # text-embeddings-ada-002 Make sure this to use your deployment name for the model api_version=azure_openai_api_version, # typically 2023-07-01-preview )
> 
> # The latest version of RAGAs is looking from ground_truth rather than ground_truths
> # and it should just be a string rather than a sequence of strings
> final_df = df_eval[['question','ground_truths','answer','contexts']]
> 
> dataset = Dataset.from_pandas(final_df)
> 
> metrics = [ faithfulness, answer_relevancy, context_precision, context_recall, context_relevancy, #answer_correctness ]
> 
> result = evaluate( dataset=dataset, column_map={"question": "question", "contexts": "contexts", "answer": "answer", "ground_truth": "ground_truths"}, llm=azure_model, embeddings=azure_embeddings, metrics=metrics, is_async=True, )
> 
> final_results = result.to_pandas()
> 
> final_results

yes it is working now but shows new error after the evaluation completion
**RuntimeError: Event loop is closed
Invalid JSON response. Expected dictionary with key 'Attributed'**


Hi @shahules786 , 

ragas version : '0.1.0'
langchain : '0.1.6'
python: 3.9

### I am using below code: (Used OpenAI llm & embedding)

``` 
from ragas.metrics import (
    context_precision,
    answer_relevancy,
    faithfulness,
    context_recall,
)
from ragas.metrics.critique import harmfulness

metrics = [
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
    harmfulness,
]
from datasets import load_dataset

amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")
result = evaluate(
    amnesty_qa["eval"], metrics=metrics, llm=llm, embeddings=embeddings,is_async=True,
)
```


### **but I am getting below error:** 

```  
File "/home/.conda/envs/ragas/lib/python3.9/site-packages/langchain_core/language_models/llms.py", line 784, in _agenerate_helper
    await self._agenerate(
  File "/home/.conda/envs/ragas/lib/python3.9/site-packages/langchain_core/language_models/llms.py", line 1196, in _agenerate
    await self._acall(prompt, stop=stop, run_manager=run_manager, **kwargs)
  File "/home/.conda/envs/ragas/lib/python3.9/site-packages/langchain_core/language_models/llms.py", line 1155, in _acall
    return await run_in_executor(
  File "/home/.conda/envs/ragas/lib/python3.9/site-packages/langchain_core/runnables/config.py", line 493, in run_in_executor
    return await asyncio.get_running_loop().run_in_executor(
  File "/home/.conda/envs/ragas/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
TypeError: _call() got an unexpected keyword argument 'temperature'
```

> Hi @shahules786 ,
> 
> ragas version : '0.1.0' langchain : '0.1.6' python: 3.9
> 
> ### I am using below code: (Used OpenAI llm & embedding)
> ```
> from ragas.metrics import (
>     context_precision,
>     answer_relevancy,
>     faithfulness,
>     context_recall,
> )
> from ragas.metrics.critique import harmfulness
> 
> metrics = [
>     faithfulness,
>     answer_relevancy,
>     context_recall,
>     context_precision,
>     harmfulness,
> ]
> from datasets import load_dataset
> 
> amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")
> result = evaluate(
>     amnesty_qa["eval"], metrics=metrics, llm=llm, embeddings=embeddings,is_async=True,
> )
> ```
> 
> ### **but I am getting below error:**
> ```
> File "/home/.conda/envs/ragas/lib/python3.9/site-packages/langchain_core/language_models/llms.py", line 784, in _agenerate_helper
>     await self._agenerate(
>   File "/home/.conda/envs/ragas/lib/python3.9/site-packages/langchain_core/language_models/llms.py", line 1196, in _agenerate
>     await self._acall(prompt, stop=stop, run_manager=run_manager, **kwargs)
>   File "/home/.conda/envs/ragas/lib/python3.9/site-packages/langchain_core/language_models/llms.py", line 1155, in _acall
>     return await run_in_executor(
>   File "/home/.conda/envs/ragas/lib/python3.9/site-packages/langchain_core/runnables/config.py", line 493, in run_in_executor
>     return await asyncio.get_running_loop().run_in_executor(
>   File "/home/.conda/envs/ragas/lib/python3.9/concurrent/futures/thread.py", line 58, in run
>     result = self.fn(*self.args, **self.kwargs)
> TypeError: _call() got an unexpected keyword argument 'temperature'
> ```

I tried the evaluation with below versions and it is working fine now : 
openai.__version__ : '1.12.0'
ragas.__version__ : '0.0.22'
langchain.__version__ : '0.1.0'

There is some problem with new version of Ragas