Repository: confident-ai/deepeval
Issue: LiteLLM is breaking which mixed json_object and json_schema
Root cause: Interface Contract Mismatch

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
When using `LiteLLMModel` with `Answer Relevancy` metric, it is unable to make request to OpenAI.

**To Reproduce**
Steps to reproduce the behavior:

1. Create a sample test
```py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.metrics import GEval

def test_correctness():
    correctness_metric = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
        threshold=0.5
    )
    test_case = LLMTestCase(
        input="I have a persistent cough and fever. Should I be worried?",
        # Replace this with the actual output from your LLM application
        actual_output="A persistent cough and fever could be a viral infection or something more serious. See a doctor if symptoms worsen or don't improve in a few days.",
        expected_output="A persistent cough and fever could indicate a range of illnesses, from a mild viral infection to more serious conditions like pneumonia or COVID-19. You should seek medical attention if your symptoms worsen, persist for more than a few days, or are accompanied by difficulty breathing, chest pain, or other concerning signs."
    )
    assert_test(test_case, [correctness_metric])
```

2. Use `LiteLLMModel` to replace the default model

```diff
from deepeval import assert_test
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.metrics import GEval
+from deepeval.models import LiteLLMModel

def test_correctness():
+   litellm_model = LiteLLMModel(model="gpt-4.1")

    correctness_metric = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
+      model=litellm_model
        threshold=0.5
    )
    test_case = LLMTestCase(
        input="I have a persistent cough and fever. Should I be worried?",
        # Replace this with the actual output from your LLM application
        actual_output="A persistent cough and fever could be a viral infection or something more serious. See a doctor if symptoms worsen or don't improve in a few days.",
        expected_output="A persistent cough and fever could indicate a range of illnesses, from a mild viral infection to more serious conditions like pneumonia or COVID-19. You should seek medical attention if your symptoms worsen, persist for more than a few days, or are accompanied by difficulty breathing, chest pain, or other concerning signs."
    )
    assert_test(test_case, [correctness_metric])
```
3. Run the test


**Expected behavior**

The `LiteLLMModel` should work correctly with schema detection.

The bug is caused by these lines:
https://github.com/confident-ai/deepeval/blob/8d2b7efcfb4e5882bcdcf7d71a478547fdfade6e/deepeval/models/llms/litellm_model.py#L139-L141

The format is set to `json_object`, but `json_schema` is configured; it is not expected behavior for OpenAI.

In the `GPTModel` it will check which schema is support and use the correct version
https://github.com/confident-ai/deepeval/blob/8d2b7efcfb4e5882bcdcf7d71a478547fdfade6e/deepeval/models/llms/openai_model.py#L194-L228


**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: macOS (M3)
 - Browser Chrome
 - Version 137.0.7151.120

**Smartphone (please complete the following information):**
 - Device: [e.g. iPhone6]
 - OS: [e.g. iOS8.1]
 - Browser [e.g. stock browser, safari]
 - Version [e.g. 22]

**Additional context**

N/A


I have the same issue when using deepeval (v3.2.3) evaluate:

```
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/.venv/lib/python3.13/site-packages/deepeval/evaluate/execute.py", line 615, in a_execute_test_cases
    await asyncio.gather(*tasks)
 File "/.venv/lib/python3.13/site-packages/deepeval/evaluate/execute.py", line 425, in execute_with_semaphore
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/.venv/lib/python3.13/site-packages/deepeval/evaluate/execute.py", line 661, in a_execute_llm_test_cases
    await measure_metrics_with_indicator(
    ...<8 lines>...
    )
  File "/.venv/lib/python3.13/site-packages/deepeval/metrics/indicator.py", line 230, in measure_metrics_with_indicator
    await asyncio.gather(*tasks)
  File "/.venv/lib/python3.13/site-packages/deepeval/metrics/indicator.py", line 243, in safe_a_measure
    await metric.a_measure(
        tc, _show_indicator=False, _in_component=_in_component
    )
  File "/.venv/lib/python3.13/site-packages/deepeval/metrics/dag/dag.py", line 104, in a_measure
    await self.dag._a_execute(metric=self, test_case=test_case)
  File "/.venv/lib/python3.13/site-packages/deepeval/metrics/dag/graph.py", line 38, in _a_execute
    await asyncio.gather(
    ...<6 lines>...
    )
  File "/.venv/lib/python3.13/site-packages/deepeval/metrics/dag/nodes.py", line 351, in _a_execute
    res, cost = await metric.model.a_generate(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        prompt, schema=TaskNodeOutput
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
openai.BadRequestError: Error code: 400 - {'error': {'message': "'json_schema' is not one of ['json_object', 'text'] - 'response_format.type'", 'type': 'invalid_request_error', 'param': None, 'code': None}}
```

Openai model is gpt-4o and Azure OpenAI endpoint with API version 2025-01-01-preview.

Hey @Aaryanverma i noticed you opened a PR but closed it - did the fix not work? Personally not 100% familar with LiteLLM so any PRs are very appreciated on this

Hey @penguine-ip yeah fix didn't worked in all places, so had to close it. I managed to use azure ai module for my work instead of litellm.