Repository: confident-ai/deepeval
Issue: ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.
Root cause: Interface Contract Mismatch

**Problem:**
When I use the local Llama-7b model for faithfulness calculation, the following error occurs. Is it because this metric does not support models with smaller parameters? It only supports closed-source LLMs, such as gpt3.5?

**ERROR:**

Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.60s/it]

Traceback (most recent call last):
  File "/python3.9/site-packages/deepeval/metrics/faithfulness/faithfulness.py", line 241, in _a_generate_truths
    res: Truths = await self.model.a_generate(prompt, schema=Truths)
**TypeError: a_generate() got an unexpected keyword argument 'schema'**

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/deepeval/metrics/utils.py", line 79, in trimAndLoadJson
    return json.loads(jsonStr)
  File "/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/python3.9/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
**json.decoder.JSONDecodeError: Extra data: line 7 column 1 (char 163)**

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 19, in <module>
    faith_score = faithfulness_score(generated_list, evidence_list, model_name_or_path, device)
  File "faithfulness.py", line 51, in faithfulness_score
    metric.measure(test_case)
  File "/deepeval/metrics/faithfulness/faithfulness.py", line 59, in measure
    loop.run_until_complete(
  File "python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/deepeval/metrics/faithfulness/faithfulness.py", line 94, in a_measure
    self.truths, self.claims = await asyncio.gather(
  File "/deepeval/metrics/faithfulness/faithfulness.py", line 245, in _a_generate_truths
    data = trimAndLoadJson(res, self)
  File "/deepeval/metrics/utils.py", line 84, in trimAndLoadJson
    raise ValueError(error_str)
**ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.**


Hello, I've been experiencing the same issue, except I'm using an Azure OpenAI instance.

Have you had any luck in solving this?

I am also getting the same error with Azure OpenAI. 

Can someone help me to get rid of this error.?

Can better support for Azure Open AI be added please?

Hi,

I have same issue with faithfulness with Azure Open AI GPT-4o model.
I am using `deepeval==2.1.1`
Also the thing is that the issue does not shows up all the time with all the cases.
I managed to process some cases before.
I tried processing in batches but even for batch of 2 test cases it resulted with the same exception.

```
File ~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/indicator.py:210, in safe_a_measure(metric, tc, ignore_errors, skip_on_missing_params)
    [203](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/indicator.py:203) async def safe_a_measure(
    [204](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/indicator.py:204)     metric: Union[BaseMetric, BaseMultimodalMetric, BaseConversationalMetric],
    [205](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/indicator.py:205)     tc: Union[LLMTestCase, MLLMTestCase, ConversationalTestCase],
    [206](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/indicator.py:206)     ignore_errors: bool,
    [207](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/indicator.py:207)     skip_on_missing_params: bool,
    [208](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/indicator.py:208) ):
    [209](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/indicator.py:209)     try:
--> [210](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/indicator.py:210)         await metric.a_measure(tc, _show_indicator=False)
    [211](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/indicator.py:211)     except MissingTestCaseParamsError as e:
    [212](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/indicator.py:212)         if skip_on_missing_params:

File ~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:106, in FaithfulnessMetric.a_measure(self, test_case, _show_indicator)
    [102](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:102) self.evaluation_cost = 0 if self.using_native_model else None
    [103](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:103) with metric_progress_indicator(
    [104](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:104)     self, async_mode=True, _show_indicator=_show_indicator
    [105](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:105) ):
--> [106](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:106)     self.truths, self.claims = await asyncio.gather(
    [107](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:107)         self._a_generate_truths(test_case.retrieval_context),
    [108](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:108)         self._a_generate_claims(test_case.actual_output),
    [109](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:109)     )
    [110](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:110)     self.verdicts = await self._a_generate_verdicts()
    [111](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:111)     self.score = self._calculate_score()

File ~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:304, in Task.__wakeup(self, future)
    [302](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:302) def __wakeup(self, future):
    [303](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:303)     try:
--> [304](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:304)         future.result()
    [305](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:305)     except BaseException as exc:
    [306](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:306)         # This may also be a cancellation.
    [307](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:307)         self.__step(exc)

File ~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:232, in Task.__step(***failed resolving arguments***)
    [228](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:228) try:
    [229](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:229)     if exc is None:
    [230](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:230)         # We use the `send` method directly, because coroutines
    [231](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:231)         # don't have `__iter__` and `__next__` methods.
--> [232](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:232)         result = coro.send(None)
    [233](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:233)     else:
    [234](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/.pyenv/versions/3.10.15/lib/python3.10/asyncio/tasks.py:234)         result = coro.throw(exc)

File ~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:258, in FaithfulnessMetric._a_generate_truths(self, retrieval_context)
    [256](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:256) except TypeError:
    [257](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:257)     res = await self.model.a_generate(prompt)
--> [258](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:258)     data = trimAndLoadJson(res, self)
    [259](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/faithfulness/faithfulness.py:259)     return data["truths"]

File ~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/utils.py:253, in trimAndLoadJson(input_string, metric)
    [251](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/utils.py:251)     if metric is not None:
    [252](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/utils.py:252)         metric.error = error_str
--> [253](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/utils.py:253)     raise ValueError(error_str)
    [254](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/utils.py:254) except Exception as e:
    [255](https://file+.vscode-resource.vscode-cdn.net/Users/Documents/repos/notebooks/~/Documents/repos/.venv/lib/python3.10/site-packages/deepeval/metrics/utils.py:255)     raise Exception(f"An unexpected error occurred: {str(e)}")

ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.
```

OK, I have found the solution that worked for me at least. It is not perfect but at lease allows to proceed.
It is basically the `truths_extraction_limit` parameter in `FaithfulnessMetric(model = metric_model, async_mode=False, truths_extraction_limit=50)`.
When I printed the string which caused error `ValueError: Evaluation LLM outputted an invalid JSON` I have found out that it failed on the generating "truths" statements based on retrieved context. 
It looked as if the output from the LLM model was kind of not complete. 
I guess that the LLM model  reached its tokens limit or something like that.
Long story short - when I put constraint on max number of "truths" `truths_extraction_limit` then it worked. It occasionally still resulted with this error but it was more on different part - on the verdict preparation. But this happens much more seldom.

This still happens when using AzureOpenAI, even with gpt-4.1 models. That's ridiculous, since they support structured outputs, that's just bad implementation and even more ridiculous ValueError.

ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.

...

Switching to RAGAS.

I realised one of the reasons this error comes up is that the json generation is truncated in the middle because of the max_tokens limit. If the output is not a complete json because of truncation, this error comes up.
If you are using a custom llm, some of these errors can be solved by increasing the max_tokens limit.

Meeting the same problem...(I've tried Qwen3-30B\Qwen2.5-14B) Who can fix it?

As I found out the hard way, the support for custom LLMs and/or AzureAI models exists only on paper.. For any serious evaluations one needs access to OpenAI API, sadly.

 You're running DeepEval's latest Answer Relevancy Metric! (using gpt-oss:20b (Ol…
Error during evaluation: 1 validation error for Statements
  Invalid JSON: EOF while parsing a value at line 1 column 0 [type=json_invalid, input_value='', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid   
gpt-oss:20b model returned empty response. This is a known issue with this model's JSON output format.

> OK, I have found the solution that worked for me at least. It is not perfect but at lease allows to proceed. It is basically the `truths_extraction_limit` parameter in `FaithfulnessMetric(model = metric_model, async_mode=False, truths_extraction_limit=50)`. When I printed the string which caused error `ValueError: Evaluation LLM outputted an invalid JSON` I have found out that it failed on the generating "truths" statements based on retrieved context. It looked as if the output from the LLM model was kind of not complete. I guess that the LLM model reached its tokens limit or something like that. Long story short - when I put constraint on max number of "truths" `truths_extraction_limit` then it worked. It occasionally still resulted with this error but it was more on different part - on the verdict preparation. But this happens much more seldom.

Yeah, I ran into the same thing. I tested this suggestion and it reduces fails (but it still happens sometimes)… Basically, from what I understand, this error occurs when the model output gets cut off or adds extra text, so DeepEval can’t parse the JSON.

**Stuff I tested that might help:**
- Use a model that handles structured JSON (like gpt-4o-2024-11-20).
- Increase max_tokens so the output isn’t truncated.
- For FaithfulnessMetric and other metrics like this, lower truths_extraction_limit (5-10) to keep outputs short.
- Avoid models that are messy with JSON.

It’s not 100% perfect, but makes the errors way less frequent.
I’m still using it, so if anyone has a suggestion for a definitive fix, please let me know!

I have an update on this:

I was using `from deepeval.models import AzureOpenAIModel` with `gpt-4o-2024-11-20`, which is a model that supports structured outputs.

DeepEval maintains lists of models that support different behaviors, as shown here:
https://github.com/confident-ai/deepeval/blob/e08eafacfd12da028a336bce5a69028a16883efd/deepeval/models/llms/openai_model.py#L88

When creating the `AzureOpenAIModel`, I was not providing the model_name parameter. Because of this, in the `a_generate` function, the model name was not being detected, so the logic defaulted to assuming that the model did not support structured outputs. 
https://github.com/confident-ai/deepeval/blob/ab650f56a43830a23836b349923cd7fef13e3a4f/deepeval/models/llms/azure_model.py#L143

This caused malformed JSONs and the error:

`ValueError Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.`

We identified that the missing `model_name` parameter in `AzureOpenAIModel` was the root cause. While this parameter is technically optional in DeepEval, it should be treated as required since it determines whether the model can handle structured outputs. After adding this parameter, the issue was resolved.

After fixing that, we ran into a new error: OpenAI’s LengthFinishReasonError. It turned out that the generations were exceeding the maximum token limit, but that’s another issue.

Hope this helps anyone running into the same problem!

I get this error intermittently when using `gpt-5-mini` as the Judge LLM (the `model` argument for [G-Eval](https://deepeval.com/docs/metrics-llm-evals#usage)). This is using DeepEval version 3.4.9.

```
ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.
```

Same error here when using gpt-5-nano and deepeval version 3.6.7.

```
ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.
``` 

@JingsenZhang Thank you for reporting this, and thank you to all those that documented your observations. I will be investigating this and will let you know once a fix is on its way.

@plopezamaya @JoeMountWSWS thanks for the reports. DeepEval’s metrics and model interface have been significantly refactored.

Could you please upgrade to the latest DeepEval (currently `deepeval==3.7.6`) and confirm whether you can still reproduce?  

If it still happens, could you share a minimal repro script and include the following details?

- Confirm DeepEval version (`python -c "import deepeval; print(deepeval.__version__)"`)
- Python version
- Which evaluation model/provider and metrics
- Full traceback (and if possible, the raw “invalid JSON” model output)

Thank you for your time on this, it is appreciated.