Repository: confident-ai/deepeval
Issue: Structured output for knowledge retention metric
Root cause: Interface Contract Mismatch

**Describe the bug**
Using structured output with schema does not work properly for the knowledge retention metric and the schema defined and passed in for it.

**To Reproduce**
Steps to reproduce the behavior:
1. Set up a DeepEval KnowledgeRetentionMetric with a custom LLM wrapper that supports structured output
2. Enable structured output by passing the schema argument to the LLM's generate method
3. Run the knowledge retention evaluation on conversational data
4. See ValidationError: "1 validation error for Knowledge data Field required"

**Expected behavior**
Model should return a response with structured output containing the evaluation result, score, and reason for the score. 

**Screenshots/Examples**
Examples of **truncated** model responses in the case where structured output is attempted by passing in the schema argument and without the schema are shown below. Different models were tested and in all cases the LLM returned an empty dictionary in the "content" field of the response when schema is used. 

Example 1: MODEL RESPONSE WITH STRUCTURED OUTPUT:  content='{}' additional_kwargs={'refusal': None} response_metadata={'token_usage': {'completion_tokens': 2, 'prompt_tokens': 670, 'total_tokens': 672, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'gpt-4.1-2025-04-14', ...

Example 2: MODEL RESPONSE WITHOUT STRUCTURED OUTPUT:  {
    "reason": "The score is 1.00 because there are no attritions, indicating no forgetfulness or loss of previously established knowledge in the LLM's responses."} content='{\n    "reason": "The score is 1.00 because there are no attritions, indicating no forgetfulness or loss  of previously established knowledge in the LLM\'s responses."\n}' additional_kwargs={'refusal': None} response_metadata={'token_usage': {'completion_tokens': 39, 'prompt_tokens': 151, 'total_tokens': 190, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'gpt-4.1-2025-04-14', ...

Ideally, the case where structured output is attempted should also include a real evaluation result in the response. 

**Desktop (please complete the following information):**
 - OS: macOS
 - Python: 3.10
 - DeepEval: Latest version
 - LangChain with Azure OpenAI
 - Model: GPT 4.1: 04-14-2025


Hey @tasneem200 can you please show the error stack? Also which DeepEval version are you on? I know for sure that a few months back not everything was structured so maybe this could be the issue

> Hey [@tasneem200](https://github.com/tasneem200) can you please show the error stack? Also which DeepEval version are you on? I know for sure that a few months back not everything was structured so maybe this could be the issue

I was on version 3.6.6, but even after upgrading to 3.7.2 the error persists. This is the error stack:

```
      ╭────────────────── Traceback (most recent call last) ───────────────────╮                     
                             │ /Users/abdalla.t/ds-cf-genai-eval/src/genai_eval/evaluators/llm_eval_d │                     
                             │ eepeval.py:1446 in _calc_metric                                        │                     
                             │                                                                        │                     
                             │   1443 │   │   result = []                                             │                     
                             │   1444 │   │   for test_case in eval_data:                             │                     
                             │   1445 │   │   │   try:                                                │                     
                             │ ❱ 1446 │   │   │   │   metric.measure(test_case)                       │                     
                             │   1447 │   │   │   except Exception as e:                              │                     
                             │   1448 │   │   │   │   default_score, error_msg = self._handle_evaluat │                     
                             │   1449 │   │   │   │   metric.score = default_score                    │                     
                             │                                                                        │                     
                             │ /Users/abdalla.t/ds-cf-genai-eval/.venv/lib/python3.10/site-packages/d │                     
                             │ eepeval/metrics/knowledge_retention/knowledge_retention.py:63 in       │                     
                             │ measure                                                                │                     
                             │                                                                        │                     
                             │    60 │   │   ):                                                       │                     
                             │    61 │   │   │   if self.async_mode:                                  │                     
                             │    62 │   │   │   │   loop = get_or_create_event_loop()                │                     
                             │ ❱  63 │   │   │   │   loop.run_until_complete(                         │                     
                             │    64 │   │   │   │   │   self.a_measure(                              │                     
                             │    65 │   │   │   │   │   │   test_case,                               │                     
                             │    66 │   │   │   │   │   │   _show_indicator=False,                   │                     
                             │                                                                        │                     
                             │ /opt/homebrew/Cellar/python@3.10/3.10.18/Frameworks/Python.framework/V │                     
                             │ ersions/3.10/lib/python3.10/asyncio/base_events.py:649 in              │                     
                             │ run_until_complete                                                     │                     
                             │                                                                        │                     
                             │    646 │   │   if not future.done():                                   │                     
                             │    647 │   │   │   raise RuntimeError('Event loop stopped before Futur │                     
                             │    648 │   │                                                           │                     
                             │ ❱  649 │   │   return future.result()                                  │                     
                             │    650 │                                                               │                     
                             │    651 │   def stop(self):                                             │                     
                             │    652 │   │   """Stop running the event loop.                         │                     
                             │                                                                        │                     
                             │ /Users/abdalla.t/ds-cf-genai-eval/.venv/lib/python3.10/site-packages/d │                     
                             │ eepeval/metrics/knowledge_retention/knowledge_retention.py:115 in      │                     
                             │ a_measure                                                              │                     
                             │                                                                        │                     
                             │   112 │   │   │   _in_component=_in_component,                         │                     
                             │   113 │   │   ):                                                       │                     
                             │   114 │   │   │   self.knowledges: List[Union[Knowledge, None]] = (    │                     
                             │ ❱ 115 │   │   │   │   await self._a_generate_knowledges(test_case.turn │                     
                             │   116 │   │   │   )                                                    │                     
                             │   117 │   │   │   self.verdicts: List[KnowledgeRetentionVerdict] = (   │                     
                             │   118 │   │   │   │   await self._a_generate_verdicts(test_case.turns) │                     
                             │                                                                        │                     
                             │ /Users/abdalla.t/ds-cf-genai-eval/.venv/lib/python3.10/site-packages/d │                     
                             │ eepeval/metrics/knowledge_retention/knowledge_retention.py:299 in      │                     
                             │ _a_generate_knowledges                                                 │                     
                             │                                                                        │                     
                             │   296 │   │   │   │   knowledges[i] = Knowledge(data=data)             │                     
                             │   297 │   │   │   else:                                                │                     
                             │   298 │   │   │   │   try:                                             │                     
                             │ ❱ 299 │   │   │   │   │   knowledges[i] = await self.model.a_generate( │                     
                             │   300 │   │   │   │   │   │   prompt, schema=Knowledge                 │                     
                             │   301 │   │   │   │   │   )                                            │                     
                             │   302 │   │   │   │   except TypeError:                                │                     
                             │                                                                        │                     
                             │ /Users/abdalla.t/ds-cf-genai-eval/src/genai_eval/evaluators/llm_eval_d │                     
                             │ eepeval.py:192 in a_generate                                           │                     
                             │                                                                        │                     
                             │    189 │   │   │   │   #     raise TypeError("Knowledge retention metr │                     
                             │        structured output schema.")                                     │                     
                             │    190 │   │   │   chat_model = base_model.with_structured_output(sche │                     
                             │        method="function_calling")                                      │                     
                             │    191 │   │                                                           │                     
                             │ ❱  192 │   │   response = await chat_model.ainvoke(prompt)             │                     
                             │    193 │   │                                                           │                     
                             │    194 │   │   if schema is not None:                                  │                     
                             │    195 │   │   │   return response                                     │                     
                             │                                                                        │                     
                             │ /Users/abdalla.t/ds-cf-genai-eval/.venv/lib/python3.10/site-packages/l │                     
                             │ angchain_core/runnables/base.py:3291 in ainvoke                        │                     
                             │                                                                        │                     
                             │   3288 │   │   │   │   │   │   part = functools.partial(step.ainvoke,  │                     
                             │   3289 │   │   │   │   │   else:                                       │                     
                             │   3290 │   │   │   │   │   │   part = functools.partial(step.ainvoke,  │                     
                             │ ❱ 3291 │   │   │   │   │   input_ = await coro_with_context(part(), co │                     
                             │   3292 │   │   │   # finish the root run                               │                     
                             │   3293 │   │   except BaseException as e:                              │                     
                             │   3294 │   │   │   await run_manager.on_chain_error(e)                 │                     
                             │                                                                        │                     
                             │ /Users/abdalla.t/ds-cf-genai-eval/.venv/lib/python3.10/site-packages/l │                     
                             │ angchain_core/output_parsers/base.py:223 in ainvoke                    │                     
                             │                                                                        │                     
                             │   220 │   │   **kwargs: Optional[Any],                                 │                     
                             │   221 │   ) -> T:                                                      │                     
                             │   222 │   │   if isinstance(input, BaseMessage):                       │                     
                             │ ❱ 223 │   │   │   return await self._acall_with_config(                │                     
                             │   224 │   │   │   │   lambda inner_input: self.aparse_result(          │                     
                             │   225 │   │   │   │   │   [ChatGeneration(message=inner_input)]        │                     
                             │   226 │   │   │   │   ),                                               │                     
                             │                                                                        │                     
                             │ /Users/abdalla.t/ds-cf-genai-eval/.venv/lib/python3.10/site-packages/l │                     
                             │ angchain_core/runnables/base.py:2146 in _acall_with_config             │                     
                             │                                                                        │                     
                             │   2143 │   │   │   │   coro = acall_func_with_variable_args(           │                     
                             │   2144 │   │   │   │   │   func, input_, config, run_manager, **kwargs │                     
                             │   2145 │   │   │   │   )                                               │                     
                             │ ❱ 2146 │   │   │   │   output: Output = await coro_with_context(coro,  │                     
                             │   2147 │   │   except BaseException as e:                              │                     
                             │   2148 │   │   │   await run_manager.on_chain_error(e)                 │                     
                             │   2149 │   │   │   raise                                               │                     
                             │                                                                        │                     
                             │ /Users/abdalla.t/ds-cf-genai-eval/.venv/lib/python3.10/site-packages/l │                     
                             │ angchain_core/output_parsers/base.py:284 in aparse_result              │                     
                             │                                                                        │                     
                             │   281 │   │   Returns:                                                 │                     
                             │   282 │   │   │   Structured output.                                   │                     
                             │   283 │   │   """                                                      │                     
                             │ ❱ 284 │   │   return await run_in_executor(None, self.parse_result, re │                     
                             │   285 │                                                                │                     
                             │   286 │   async def aparse(self, text: str) -> T:                      │                     
                             │   287 │   │   """Async parse a single string model output into some st │                     
                             │                                                                        │                     
                             │ /Users/abdalla.t/ds-cf-genai-eval/.venv/lib/python3.10/site-packages/l │                     
                             │ angchain_core/runnables/config.py:611 in run_in_executor               │                     
                             │                                                                        │                     
                             │   608 │                                                                │                     
                             │   609 │   if executor_or_config is None or isinstance(executor_or_conf │                     
                             │   610 │   │   # Use default executor with context copied from current  │                     
                             │ ❱ 611 │   │   return await asyncio.get_running_loop().run_in_executor( │                     
                             │   612 │   │   │   None,                                                │                     
                             │   613 │   │   │   cast("Callable[..., T]", partial(copy_context().run, │                     
                             │   614 │   │   )                                                        │                     
                             │                                                                        │                     
                             │ /opt/homebrew/Cellar/python@3.10/3.10.18/Frameworks/Python.framework/V │                     
                             │ ersions/3.10/lib/python3.10/concurrent/futures/thread.py:58 in run     │                     
                             │                                                                        │                     
                             │    55 │   │   │   return                                               │                     
                             │    56 │   │                                                            │                     
                             │    57 │   │   try:                                                     │                     
                             │ ❱  58 │   │   │   result = self.fn(*self.args, **self.kwargs)          │                     
                             │    59 │   │   except BaseException as exc:                             │                     
                             │    60 │   │   │   self.future.set_exception(exc)                       │                     
                             │    61 │   │   │   # Break a reference cycle with the exception 'exc'   │                     
                             │                                                                        │                     
                             │ /Users/abdalla.t/ds-cf-genai-eval/.venv/lib/python3.10/site-packages/l │                     
                             │ angchain_core/runnables/config.py:602 in wrapper                       │                     
                             │                                                                        │                     
                             │   599 │                                                                │                     
                             │   600 │   def wrapper() -> T:                                          │                     
                             │   601 │   │   try:                                                     │                     
                             │ ❱ 602 │   │   │   return func(*args, **kwargs)                         │                     
                             │   603 │   │   except StopIteration as exc:                             │                     
                             │   604 │   │   │   # StopIteration can't be set on an asyncio.Future    │                     
                             │   605 │   │   │   # it raises a TypeError and leaves the Future pendin │                     
                             │                                                                        │                     
                             │ /Users/abdalla.t/ds-cf-genai-eval/.venv/lib/python3.10/site-packages/l │                     
                             │ angchain_core/output_parsers/openai_tools.py:343 in parse_result       │                     
                             │                                                                        │                     
                             │   340 │   │   │   │   )                                                │                     
                             │   341 │   │   │   │   raise ValueError(msg)                            │                     
                             │   342 │   │   │   try:                                                 │                     
                             │ ❱ 343 │   │   │   │   pydantic_objects.append(name_dict[res["type"]](* │                     
                             │   344 │   │   │   except (ValidationError, ValueError):                │                     
                             │   345 │   │   │   │   if partial:                                      │                     
                             │   346 │   │   │   │   │   continue                                     │                     
                             │                                                                        │                     
                             │ /Users/abdalla.t/ds-cf-genai-eval/.venv/lib/python3.10/site-packages/p │                     
                             │ ydantic/main.py:250 in __init__                                        │                     
                             │                                                                        │                     
                             │    247 │   │   """                                                     │                     
                             │    248 │   │   # `__tracebackhide__` tells pytest and some other tools │                     
                             │        from tracebacks                                                 │                     
                             │    249 │   │   __tracebackhide__ = True                                │                     
                             │ ❱  250 │   │   validated_self = self.__pydantic_validator__.validate_p │                     
                             │        self_instance=self)                                             │                     
                             │    251 │   │   if self is not validated_self:                          │                     
                             │    252 │   │   │   warnings.warn(                                      │                     
                             │    253 │   │   │   │   'A custom validator is returning a value other  │                     
                             ╰────────────────────────────────────────────────────────────────────────╯                     
                             ValidationError: 1 validation error for Knowledge                                              
                             data                                                                                           
                               Field required [type=missing, input_value={}, input_type=dict]                    
```

And this is how the generate method is calling with structured output: 

```
chat_model = model.with_structured_output(schema, method="function_calling")
response = await chat_model.ainvoke(prompt)
```

@tasneem200, thank you for reporting this. I will be looking into it and will let you know if I have further questions.

@tasneem200 thanks again for reporting this!

I just committed a fix for the DeepEval side of this in the linked PR, specifically this commit:

https://github.com/confident-ai/deepeval/pull/2367/commits/6265ba8f2c34bbfe06d5653f621938786b4307a2

The change makes `KnowledgeRetentionMetric` more robust when used with structured output:
- the `Knowledge` schema no longer raises a `Field required` error when the model returns an empty `{}` for that schema
- the metric now simply treats such cases as "no extracted knowledge" instead of crashing

So once that PR is merged and released, you shouldn’t see the `ValidationError` anymore. If your provider or LangChain/Azure setup still returns `{}` for the structured schema, the metric will run successfully but will effectively see that as "no data", so it’s worth double checking your custom model configuration if that persists.

all the best!