Repository: confident-ai/deepeval
Issue: G-Eval model `gpt-5-mini` errors
Root cause: External Dependency Breakage

**Describe the bug**

I encounter intermittent errors of various kinds when attemping to use `gpt-5-mini` as the judge LLM. Curiously, this doesn't happen when the completions I'm evaluating are generated by `gpt-5` or `gpt-5-nano`, only `gpt-5-mini` so perhaps these errors occur when a model is the judge LLM for itself?

```python
deepeval.metrics.GEval(
  ...
  model="gpt-5-mini",
  ...
)
```

```
ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.
```

```
            # don't use log probabilities for unsupported gpt models
            if no_log_prob_support(self.model):
>               raise AttributeError("log_probs unsupported.")
E               AttributeError: log_probs unsupported.

/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/site-packages/deepeval/metrics/g_eval/g_eval.py:266: AttributeError
```

```
        try:
>           obj, end = self.scan_once(s, idx)
                       ^^^^^^^^^^^^^^^^^^^^^^
E           json.decoder.JSONDecodeError: Invalid \escape: line 3 column 150 (char 165)

/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/json/decoder.py:354: JSONDecodeError
```

```
        try:
            return json.loads(jsonStr)
        except json.JSONDecodeError:
            error_str = "Evaluation LLM outputted an invalid JSON. Please use a better evaluation model."
            if metric is not None:
                metric.error = error_str
>           raise ValueError(error_str)
E           ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.

/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/site-packages/deepeval/metrics/utils.py:378: ValueError
```

**Expected behavior**

I expect the G-Eval metric to work, as it does for `gpt-4o-mini`, which is presumably a weaker evaluation model than `gpt-5-mini`.

Also, this model is listed here: https://deepeval.com/integrations/models/openai

**Environment**

This is running on GitHub Actions
- OS: Ubuntu 24 LTS
- DeepEval Version: 3.6.9


@JoeMountWSWS Chatgpt 5 mini doesn't support logs_prob as it is mentioned [here](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/reasoning?utm_source=chatgpt.com&tabs=gpt-5%2Cpython%2Cpy) and in the error that you have posted. For G-Eval to work it needs log_prob. So OpenAI needs to fix this, not deepeval. As for json response you can force the model to give json output. As primarily 5-mini doesn't give structured response everytime, like how gpt-4o-mini gives.

> [@JoeMountWSWS](https://github.com/JoeMountWSWS) Chatgpt 5 mini doesn't support logs_prob as it is mentioned [here](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/reasoning?utm_source=chatgpt.com&tabs=gpt-5%2Cpython%2Cpy) and in the error that you have posted. For G-Eval to work it needs log_prob. So OpenAI needs to fix this, not deepeval. As for json response you can force the model to give json output. As primarily 5-mini doesn't give structured response everytime, like how gpt-4o-mini gives.

Thank you Nitish.

Please be aware that `gpt-5-mini` is listed on the DeepEval OpenAI model page here: https://deepeval.com/integrations/models/openai which led me to believe it was compatible with DeepEval's `GEval` metric.

Also, as I explained in this issue above, `gpt-5-mini` _does_ work when used as a judge LLM evaluating completions generated by `gpt-5` and `gpt-5-nano`, but for some reason it crashed when evaluating those generated by `gpt-5-mini`. At least, that was my experience recently. I haven't tested this rigorously.

Hey @JoeMountWSWS @Nitish-Kshatriya-P is 100% spot on for this one, I'd also recommend to not use mini models for evaluation regardless.

> Hey [@JoeMountWSWS](https://github.com/JoeMountWSWS) [@Nitish-Kshatriya-P](https://github.com/Nitish-Kshatriya-P) is 100% spot on for this one, I'd also recommend to not use mini models for evaluation regardless.

Thanks Jeffrey. What size of model is best to use? The biggest ones we can afford?

Also, do you recommend using a different kind of model from the one used in our AI system? E.g. if we use ChatGPT inside our chatbot, should we use Gemini or Claude models to evaluate as the judge LLM?