Repository: confident-ai/deepeval
Issue: Schema Error when using Vertexai Custom LLM Evaluation
Root cause: Interface Contract Mismatch

I get an error when I try to use instructor for a customLLM (VertexAI not gemini api). 

**Code:**
#@title Custom LLM for DeepEval with JSON output enforcer

from pydantic import BaseModel
import instructor

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric, ContextualRelevancyMetric

from deepeval.models import DeepEvalBaseLLM
from vertexai.generative_models import GenerativeModel, HarmCategory, HarmBlockThreshold,  SafetySetting

class GoogleVertexAI_Json(DeepEvalBaseLLM):
    """Class that implements Vertex AI for DeepEval"""
    def __init__(self, model_name, *args, **kwargs):
        super().__init__(model_name, *args, **kwargs)

    def load_model(self, *args, **kwargs):
        # Initialize safety filters for Vertex AI model
        # This is important to ensure no evaluation responses are blocked
        safety_settings = {
            HarmCategory.HARM_CATEGORY_UNSPECIFIED: HarmBlockThreshold.BLOCK_NONE,
            HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,
            HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,
            HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,
            HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_NONE
        }

        vertexai.init(project=PROJECT_ID, location=LOCATION)
        
        return GenerativeModel(
            model_name=self.model_name,
            safety_settings=safety_settings)

    def generate(self, prompt: str, schema: BaseModel) -> BaseModel:
        instructor_client = instructor.from_vertexai(
            client=self.load_model(),
            mode=instructor.Mode.VERTEXAI_TOOLS,
        )
        
        resp = instructor_client.messages.create(
            messages=[
                {
                    "role": "user",
                    "content": prompt,
                }
            ],
            response_model=schema,
        )
        return resp
  
    async def a_generate(self, prompt: str, schema: BaseModel) -> BaseModel:
        return self.generate(prompt, schema)

    def get_model_name(self) -> str:
        return self.model_name

EVAL_MODEL_NAME = "gemini-2.0-flash-001"

input = QUESTION

#output = result
output = result
print(f"Output: {output}")

retrieval_context = [doc.page_content for doc,score in CONTEXT]

test_case = LLMTestCase(
    input=input,
    actual_output=output,
    retrieval_context=retrieval_context
)

print(f"Evaluating with model: {EVAL_MODEL_NAME}")
eval_model = GoogleVertexAI_Json(
                        model_name=EVAL_MODEL_NAME
                                )
contextual_relevancy = ContextualRelevancyMetric(model=eval_model, include_reason=True)

eval = evaluate([test_case], [contextual_relevancy], print_results = False)

**Error:**
ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.11/dist-packages/google/protobuf/json_format.py in _ConvertScalarFieldValue(value, field, path, require_str)
    924         try:
--> 925           number = int(value)
    926           enum_value = field.enum_type.values_by_number.get(number, None)

ValueError: invalid literal for int() with base 10: 'NULL'

The above exception was the direct cause of the following exception:

EnumStringValueParseError                 Traceback (most recent call last)
48 frames
EnumStringValueParseError: Invalid enum value NULL for enum type google.cloud.aiplatform.v1beta1.Type

The above exception was the direct cause of the following exception:

EnumStringValueParseError                 Traceback (most recent call last)
EnumStringValueParseError: Invalid enum value NULL for enum type google.cloud.aiplatform.v1beta1.Type at Schema.properties[verdicts].items.properties[reason].anyOf[1].type

The above exception was the direct cause of the following exception:

ParseError                                Traceback (most recent call last)
ParseError: Failed to parse type field: Invalid enum value NULL for enum type google.cloud.aiplatform.v1beta1.Type at Schema.properties[verdicts].items.properties[reason].anyOf[1].type.

The above exception was the direct cause of the following exception:

ParseError                                Traceback (most recent call last)
ParseError: Failed to parse anyOf field: Failed to parse type field: Invalid enum value NULL for enum type google.cloud.aiplatform.v1beta1.Type at Schema.properties[verdicts].items.properties[reason].anyOf[1].type..

The above exception was the direct cause of the following exception:

ParseError                                Traceback (most recent call last)
ParseError: Failed to parse properties field: Failed to parse anyOf field: Failed to parse type field: Invalid enum value NULL for enum type google.cloud.aiplatform.v1beta1.Type at Schema.properties[verdicts].items.properties[reason].anyOf[1].type...

The above exception was the direct cause of the following exception:

ParseError                                Traceback (most recent call last)
ParseError: Failed to parse items field: Failed to parse properties field: Failed to parse anyOf field: Failed to parse type field: Invalid enum value NULL for enum type google.cloud.aiplatform.v1beta1.Type at Schema.properties[verdicts].items.properties[reason].anyOf[1].type....

The above exception was the direct cause of the following exception:

ParseError                                Traceback (most recent call last)
/usr/local/lib/python3.11/dist-packages/google/protobuf/json_format.py in _ConvertFieldValuePair(self, js, message, path)
    690       except ParseError as e:
    691         if field and field.containing_oneof is None:
--> 692           raise ParseError(
    693               'Failed to parse {0} field: {1}.'.format(name, e)
    694           ) from e

ParseError: Failed to parse properties field: Failed to parse items field: Failed to parse properties field: Failed to parse anyOf field: Failed to parse type field: Invalid enum value NULL for enum type google.cloud.aiplatform.v1beta1.Type at Schema.properties[verdicts].items.properties[reason].anyOf[1].type.....

hey @huiyi-liang can you please ask this in our discord? hard to see the error here without the formatting

Premise: I'm not using `instructor` but directly utilising the Vertex AI Python SDK API to obtain structured output. The SDK provides a straightforward way to achieve this, so preferred not relying on additional third-party dependency unnecessary. 

I encountered similar issues, and it comes down to Google Vertex AI using a simplified, stricter JSON Schema model, as outlined in their documentation:

- https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output

This mismatch caused issues when sending **some** DeepEval metric schemas directly to Vertex AI, as their validation rules are stricter. 

Example of metric schema that was causing the error:
`BiasVerdict` that, at run time, got passed as 
```
{'$defs': {'BiasVerdict': {'properties': {'verdict': {'enum': ['yes', 'no'], 'title': 'Verdict', 'type': 'string'}, 'reason': {'anyOf': [{'type': 
'string'}, {'type': 'null'}], 'default': None, 'title': 'Reason'}}, 'required': ['verdict'], 'title': 'BiasVerdict', 'type': 'object'}}, 'properties': {'verdicts':
{'items': {'$ref': '#/$defs/BiasVerdict'}, 'title': 'Verdicts', 'type': 'array'}}, 'required': ['verdicts'], 'title': 'Verdicts', 'type': 'object'}
```

While waiting for a more robust solution from DeepEval folks, I resolved this by writing a function that makes the Pydantic-generated JSON schema compatible with Vertex AI before this is passed to vertexai.generative_models.GenerativeModel in both generate_content and generate_content_async. Perhaps a similar approach could work for the `instructure` implementation too.

Thanks @exfalsoquodlibet !

I'm relatively new at this. Is there an example code or resources you can share on creating a function that makes the Pydantic-generated JSON schema compatible with Vertex AI? I'm not tied to using the `instructor` library.

@exfalsoquodlibet we mainly rely on community contributions for these custom model providers since its hard for us to keep up with everything, glad it worked though. If there's a PR you can make for the docs for example though to show how its done would really appreciate it and @huiyi-liang  can also take note of it