Repository: confident-ai/deepeval
Issue: Possible bug when using Azure Open AI model to bulk evaluate
Root cause: Environment Incompatibility

Hi, fantastic package - it's working brilliantly when I evaluate a couple of test cases but I'm running into an error when I try and evaluate in bulk.

I'm using an Azure Open AI model, I've used the example code on the confident-ai.com page to configure the deepeval environment for Azure Open AI and create a custom model for evaluation. All of this is apparently working perfectly for evaluating single test cases.

However if I then try and use evaluate() to evaluate in bulk (my use case requires running the evaluation regularly over hundreds of test cases), I run into this error 
![image](https://github.com/confident-ai/deepeval/assets/92313732/1cc0c4e7-170e-43c9-a6dd-b788214959cf)

Fairly new to python, but digging into the trace message it looks like the error is being thrown by the test_run.py. 

Full details:
- I'm working in a databricks notebook, runtime 13.3
- Python 3.10
- latest version of deepeval

Code:
# console command to configure deepeval env
`%sh deepeval set-azure-openai --openai-endpoint=<ENDPOINT>\
    --openai-api-key=<API KEY> \
    --deployment-name=<DEPLOYMENT NAME> \
    --openai-api-version=<API VERSION>\
    --model-version=<MODEL VERSION>

# create custom model
from langchain_openai import AzureChatOpenAI
from deepeval.models.base_model import DeepEvalBaseLLM

class AzureOpenAI(DeepEvalBaseLLM):
    def __init__(
        self,
        model
    ):
        self.model = model

    def load_model(self):
        return self.model

    def _call(self, prompt: str) -> str:
        chat_model = self.load_model()
        return chat_model.invoke(prompt).content

    def get_model_name(self):
        return "Custom Azure OpenAI Model"

custom_model = AzureChatOpenAI(
    openai_api_version=<API VERSION> ,
    azure_deployment=<DEPLOYMENT NAME>,
    azure_endpoint=<ENDPOINT>,
    openai_api_key=<API KEY>,
)

azure_openai = AzureOpenAI(model=custom_model)
print(azure_openai("Write me a joke"))

# create dummy test cases and evaluate
from deepeval.dataset import EvaluationDataset
from deepeval.test_case import LLMTestCase, LLMTestCaseParams 
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric, ContextualPrecisionMetric, ContextualRecallMetric, BiasMetric, ToxicityMetric, ContextualRelevancyMetric

### create dummy test cases
first_test_case = LLMTestCase(input="How big is the sun?", actual_output="really big")
second_test_case = LLMTestCase(input="What colour is grass?", actual_output="The cow said moo")
test_cases = [first_test_case, second_test_case]
dataset = EvaluationDataset(test_cases=test_cases)

### measure answer relevancy
metric = AnswerRelevancyMetric(model=azure_openai)

### evaluate bulk
dataset.evaluate([metric])
`
Returns AttributeError: 'TestRun' object has no attribute 'model_dump'

Any help you can give would be apprecaiated

Let me know if you need any other details

Joe


@jwretham hey Joe! I believe this is due to Pydantic in your environment being outdated, we see this usually in jupyter environments.

Basically, when executing test cases we write them to a temporary file since for some modes of execution we run things in parallel on different processes, and here is the line of code where it errored for you during this process: https://github.com/confident-ai/deepeval/pull/new/features/gevalasync

Can you please let me know which version of pydantic are you on? Also found this answer on SOF: https://stackoverflow.com/questions/77251901/pydantic-basemodel-model-dump-throughs-attributeerror

which outlines how to fix it if you find it to be outdated. Let me know how it goes :)


Hi, penguine-ip, thank you for the quick response.

You were absolutely right - the version of Pydantic I was using 1.10.6. However if I run this (based on the SOF):

pip uninstall pydantic
pip uninstall pydantic
pip install -U pydantic

It updates to 2.6.3 and evaluate now runs - fantastic.

It has highlighted anther problem however, as soon as I get above a 2/3 test cases the console output messages printed for each test case (i.e. below) begin to make the notebook unstable and unusable 
![image](https://github.com/confident-ai/deepeval/assets/92313732/ad3f5628-fcc8-4b23-ba93-b61d2ec643e1)

Is there any way to disable these console messages? I'm capturing the score/reasoning in a list so don't need the readout

Apologies if I'm missing how to do this in the docs

Thanks for your help
 

@jwretham not at the moment, but i can make it part of the release to disable these. I'm also introducing async execution to deepeval tomorrow, there might be some bugs that I haven't caught but it should help speed up test case evaluation a lot. Please let me know if you find any bugs 

@jwretham also come join our discord, there will be major updates over the weekend and any breaking changes will be updated there: https://discord.com/invite/a3K9c8GRGt

Amazing - I'll experiment with the new release when I'm back in the office next week and let you know if I encounter any issues.

Thanks again for your help with this - and for putting a great package together

@jwretham No worries! Also saw you in discord. The new update is out: v0.20.85, any bugs just let me know and it will be fixed. Thanks!

@jwretham also the docs for evaluate function to not print outputs: https://docs.confident-ai.com/docs/evaluation-introduction#evaluating-without-pytest