zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Add support for user parameter in AzureOpenAI integration
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I'm trying to use DeepEval but the way our AzureOpenAI instance is configured requires the user parameter which can't be sent using DeepEval integration. Using the integration right now results in a 422 status code error.

**Describe the solution you'd like**
[LlamaIndex](https://docs.llamaindex.ai/en/stable/api_reference/llms/azure_openai/) supports sending additional parameters using the `additional_kwargs` parameter. Something similar would be a good solution to send parameters needed based on their [API reference](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/reference#chat-completions:%7E:text=the%20relevant%20token.-,user,-string)

**Describe alternatives you've considered**
At this point there doesn't look to be a workaround for DeepEval, only way could be creating a custom LLM function for the ground up, but would try to avoid add if this could be fit in in the existing integration.

**Additional context**
Add any other context or screenshots about the feature request here.


Hey @bojackhorseman0309 would you be able to make a quick PR on this? Team's pretty full right now in terms of workload but recognize this is important and do want to have this in deepeval, thank you in advnce

Sure @penguine-ip will give it a go. Thanks
Repository: confident-ai/deepeval
Issue: Is xai_sdk required as a dependency
Root cause: Environment Incompatibility

**Describe the bug**
When running the deepeval command, I encounter a ModuleNotFoundError because the xai_sdk module is not found. This occurs during the import of GrokModel in deepeval.models.llms.grok_model. The xai_sdk module appears to be a dependency for GrokModel, but it’s not installed automatically, nor is it documented as a requirement.

**To Reproduce**
Steps to Reproduce:

* Create a Python 3.10 virtual environment: python3.10 -m venv .venv
* Activate the environment: source .venv/bin/activate
* Install deepeval: pip install deepeval
* Run a script that has deepeval deepeval
* Observe the ModuleNotFoundError: No module named 'xai_sdk'.

**Expected behavior**
Expected Behavior:

The deepeval import statement should run without errors.
If xai_sdk is a required dependency, it should be installed automatically via pip install deepeval.
If xai_sdk is optional (e.g., only for GrokModel), the documentation should clearly state how to install it or how to skip GrokModel functionality.

**Desktop (please complete the following information):**
 - OS: ubuntu
 - Version latest

**Additional context**
```
Traceback (most recent call last):
  File "[REDACTED]/.venv/bin/deepeval", line 4, in <module>
    from deepeval.cli.main import app
  File "[REDACTED]/.venv/lib/python3.10/site-packages/deepeval/__init__.py", line 9, in <module>
    from deepeval.evaluate import evaluate, assert_test, test_run
  File "[REDACTED]/.venv/lib/python3.10/site-packages/deepeval/evaluate/__init__.py", line 1, in <module>
    from .evaluate import evaluate, assert_test, dataset
  File "[REDACTED]/.venv/lib/python3.10/site-packages/deepeval/evaluate/evaluate.py", line 15, in <module>
    from deepeval.dataset.dataset import EvaluationDataset
  File "[REDACTED]/.venv/lib/python3.10/site-packages/deepeval/dataset/__init__.py", line 1, in <module>
    from .dataset import EvaluationDataset
  File "[REDACTED]/.venv/lib/python3.10/site-packages/deepeval/dataset/dataset.py", line 14, in <module>
    from deepeval.metrics import (
  File "[REDACTED]/.venv/lib/python3.10/site-packages/deepeval/metrics/__init__.py", line 1, in <module>
    from .base_metric import (
  File "[REDACTED]/.venv/lib/python3.10/site-packages/deepeval/metrics/base_metric.py", line 11, in <module>
    from deepeval.models import DeepEvalBaseLLM
  File "[REDACTED]/.venv/lib/python3.10/site-packages/deepeval/models/__init__.py", line 7, in <module>
    from deepeval.models.llms import (
  File "[REDACTED]/.venv/lib/python3.10/site-packages/deepeval/models/llms/__init__.py", line 10, in <module>
    from .grok_model import GrokModel
  File "[REDACTED]/.venv/lib/python3.10/site-packages/deepeval/models/llms/grok_model.py", line 3, in <module>
    from xai_sdk import Client, AsyncClient
ModuleNotFoundError: No module named 'xai_sdk'
Error: Process completed with exit code 1.
```

Seems like this issue appeared in version `3.3.1`, but is fixed in `3.3.2`.

Maybe try updating your version of deepeval.

Hey @DaStru yes it is fixed @aadityapattabhiraman :)
Repository: confident-ai/deepeval
Issue: O1 Models Incompatible with ArenaGEval due to response_format Parameter
Root cause: Interface Contract Mismatch


  ❗BEFORE YOU BEGIN❗
  Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

  Describe the bug
  O1 models (o1, o1-2024-12-17, o3-mini, etc.) fail when used with ArenaGEval because DeepEval internally passes a response_format parameter with type json_schema, which O1 models don't support.

  To Reproduce
  Steps to reproduce the behavior:
  1. Create an O1 model: gpt_judge = GPTModel(model="o1")
  2. Create an ArenaGEval metric with the O1 model: arena_metric = ArenaGEval(name="test", criteria="Compare outputs", model=gpt_judge)
  3. Create test cases and arena test case with contestants
  4. Call await arena_metric.a_measure(arena_test_case)
  5. See error: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'temperature' is not supported with this model.\", 'type': 'invalid_request_error', 'param': 'temperature', 'code': 'u...",


  Expected behavior
  O1 models should work with ArenaGEval, either by automatically detecting that the model doesn't support structured output and falling back to text parsing, or by providing a configuration option to disable structured output. Please also add support for O3

  Desktop (please complete the following information):
  - OS: Linux Ubuntu
  - DeepEval Version: Latest
  - Python Version: 3.11

As with https://github.com/confident-ai/deepeval/issues/1821, I'm unable to reproduce the issue after following the above steps with o1. Have you tried updating to the last version of DeepEval? Would help if you could post the error trace here if you are still unable to resolve the issue.

I also see that there is no mention of evaluation_params here.
Repository: confident-ai/deepeval
Issue: GeminiModel Serialization Error with Pydantic ModelMetaclass
Root cause: Interface Contract Mismatch

❗BEFORE YOU BEGIN❗
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

  Describe the bug
  GeminiModel instances cause serialization failures when used with ArenaGEval due to Pydantic's ModelMetaclass not being serializable. This appears to be related to internal serialization within DeepEval.

  To Reproduce
  Steps to reproduce the behavior:
  1. Create a GeminiModel: gemini_judge = GeminiModel(model_name="gemini-2.5-pro", api_key="YOUR_KEY", temperature=0)
  2. Create an ArenaGEval metric with the Gemini model: arena_metric = ArenaGEval(name="test", criteria="Compare outputs", model=gemini_judge)
  3. Create test cases and arena test case with contestants
  4. Call await arena_metric.a_measure(arena_test_case)
  5. See error: Unable to serialize unknown type: <class 'pydantic._internal._model_construction.ModelMetaclass'>

  Expected behavior
  GeminiModel should work seamlessly with ArenaGEval without serialization errors.

  Desktop (please complete the following information):
  - OS: Linux Ubuntu
  - DeepEval Version: Latest
  - Python Version: 3.11
  - Pydantic Version: 2.x

  Additional context
  This appears to be related to Pydantic v2's internal model construction metaclass not being serializable. The error occurs during ArenaGEval's internal processing, likely when DeepEval tries to serialize the model object. Similar issue discussed in Pydantic
  GitHub: https://github.com/pydantic/pydantic/issues/10420

  The issue prevents using Gemini models as judges in arena evaluations. AnthropicModel and GPTModel (for non-O1 models) work fine, suggesting this is specific to the GeminiModel implementation.

  Both GPTModel (non-O1) and AnthropicModel work correctly with ArenaGEval, so this appears to be specific to these two model types.

Hey @arkokoley. I was unable to reproduce this error with your steps above. Do you have an error trace that you can post here? Maybe your LLM test case parameters are also error-causing, although this is unlikely since you mentioned this error is specific to Gemini.
Repository: confident-ai/deepeval
Issue: args&kwargs are not used in `load_model()`
Root cause: Validation Gap

**Describe the bug**
Code comments claim `# args and kwargs will be passed to the underlying model, in load_model function`, but this is not the case. This leads to issues when for example the timeout is not set high enough, but cannot be adjusted using kwargs, since they are never sent to the LLM initialization.

**To Reproduce**
I only checked `openai_model.py` and `azure_model.py`, but both do not utilize `self.args` nor `self.kwargs`. Maybe I missed something?

**Expected behavior**
 `self.args` and `self.kwargs` are passed to model instantiation.


@simon376 will be fixing it soon!

@simon376 for our OpenAI integration, I think this isn't currently supported, but I believe `self.kwargs` is used in Azure OpenAI, specifically:

https://github.com/confident-ai/deepeval/blob/5bd7aaf05cda0689c1720d318de8c44ce9fb1522/deepeval/models/llms/azure_model.py#L276

^Is this what you were looking for?

You're right, that's what I was looking for. Seems like I was not on the latest release, since the fix was only included 12d ago, my bad.

@simon376 Just made a PR to add *kwarg support for the remaining model providers: https://github.com/confident-ai/deepeval/pull/1827
Repository: confident-ai/deepeval
Issue: Recursive invoking secure_exec
Root cause: Algorithmic Error

```
# https://github.com/confident-ai/deepeval/blob/c7a6784d63af8f2a2c3510c404cfcddd56999c84/deepeval/benchmarks/human_eval/human_eval.py#L63

def secure_exec(code_str, global_vars=None, local_vars=None):
    """Securely execute code with restricted globals and locals."""
    if global_vars is None:
        global_vars = {}
    if local_vars is None:
        local_vars = {}

    # Create a restricted globals dictionary with only safe built-ins
    safe_globals = {
        "__builtins__": {
            "abs": abs,
            "all": all,
            "any": any,
            "bin": bin,
            "bool": bool,
            "chr": chr,
            "dict": dict,
            "enumerate": enumerate,
            "filter": filter,
            "float": float,
            "hex": hex,
            "int": int,
            "len": len,
            "list": list,
            "map": map,
            "max": max,
            "min": min,
            "oct": oct,
            "ord": ord,
            "pow": pow,
            "range": range,
            "reversed": reversed,
            "round": round,
            "set": set,
            "sorted": sorted,
            "str": str,
            "sum": sum,
            "tuple": tuple,
            "zip": zip,
            "Exception": Exception,
            "ValueError": ValueError,
            "TypeError": TypeError,
            "IndexError": IndexError,
            "KeyError": KeyError,
        }
    }
    safe_globals.update(global_vars)

    try:
        # Compile the code first to validate syntax
        compiled_code = compile(code_str, "<string>", "exec")
        # Execute with restricted environment
        secure_exec(compiled_code, safe_globals, local_vars)
        return local_vars
    except Exception as e:
        raise e
```

The code here recursively inokes itself.  Line63 should be replaced with `exec(compiled_code, safe_globals, local_vars)`

@superboy-zjc Fixed! Thanks for raising the issue!
Repository: confident-ai/deepeval
Issue: Verbose logs in metrics does nothing
Root cause: Validation Gap

Hi!
I was trying to switch off the verbose logs while running the RAG metrics evaluations and discovered that `verbose_mode` argument is obsolete and not used anywhere in the code, at least for the following metrics:
- Answer Relevancy
- Faithfulness
- Contextual Precision
- Contextual Recall
- Contextual Relevancy 

My expectations going through the docs was that this argument controls the verbose_logs, so it would be good to either follow this behaviour or remove the argument completely.

Hey @akornyukhin `verbose_mode` is used here: https://github.com/confident-ai/deepeval/blob/874b18cab2b988387580cd6ba5c805e84a158370/deepeval/metrics/utils.py#L151 in each metric. If it still prints after setting verbose mode to False, maybe you have verbose mode on somewhere in the `evaluate` method (if that's how you're running your evals). It overwrites whatever config you've set in the metrics.

Hey @penguine-ip, thanks for the explanation
My goal is to get rid of `verbose_logs` in the results, e.g. below

```
{
                        "name": "Faithfulness",
                        "threshold": 0.5,
                        "success": true,
                        "score": 1.0,
                        "reason": null,
                        "strict_mode": false,
                        "evaluation_model": "gpt-4.1",
                        "error": null,
                        "evaluation_cost": 0.019962,
                        "verbose_logs": "Truths (limit=....."
}
```

I think DisplayConfig controls what is being printed out during/after run?

And yes, I am running through evaluate function

```
from deepeval.metrics import (
            AnswerRelevancyMetric,
            FaithfulnessMetric,
            ContextualRelevancyMetric,
            ContextualPrecisionMetric,
            ContextualRecallMetric,
        )
        from deepeval import evaluate

        tests_results = evaluate(
            self.dataset,
            metrics=[
                AnswerRelevancyMetric(model="gpt-4.1", verbose_mode=False, include_reason=False),
                FaithfulnessMetric(model="gpt-4.1", verbose_mode=False, include_reason=False),
                ContextualPrecisionMetric(model="gpt-4.1", verbose_mode=False, include_reason=False),
                ContextualRecallMetric(model="gpt-4.1", verbose_mode=False, include_reason=False),
                ContextualRelevancyMetric(model="gpt-4.1", verbose_mode=False, include_reason=False),
            ],
        )
```

I see @akornyukhin yes `verbose_mode` just controls the printing unfortunately

Thanks @penguine-ip!
Are there any plans to get such feature in the future? 

Unfortunately no because wee delegate users to do the post data processing themselves to create whatever subset of data they require (also gives us a standard data structure to return to not confuse users)
Repository: confident-ai/deepeval
Issue: Aikido security vulnerability tool is flagging deepeval as malware
Root cause: External Dependency Breakage

Hello,

Are you folks aware that the tool Aikido https://www.aikido.dev/ is flagging `deepeval` as malware?

We've been using `deepeval` but now we are deterred from doing it because it screws our Security SLA.

<img width="608" height="700" alt="Image" src="https://github.com/user-attachments/assets/56436252-4982-46ae-8fde-ccd4b1055114" />



@sergioorbio does it say why?

Hey @sergioorbio,

I'm the security researcher at Aikido who's responsible for malware detection. 👋 

This seems like a bug on our end. We have different types of markings for packages, like `Malware`, `Telemetry`, and a few others. 

In this case, we've marked Deepeval as `Telemetry` (https://github.com/confident-ai/deepeval/blob/main/deepeval/telemetry.py), but it seems like something went wrong and it got turned into a `Malware` alert instead. We will get that fixed ASAP. 


Hey @CharlieEriksen thanks! @sergioorbio please rest assured. 

@penguine-ip below you can see the reasons shown by Aikido 

<img width="608" height="266" alt="Image" src="https://github.com/user-attachments/assets/6932ad26-0557-45cb-b434-5edfd5e7c9d3" />


Thanks @CharlieEriksen!

@sergioorbio You can turn it off 

<img width="1315" height="883" alt="Image" src="https://github.com/user-attachments/assets/a7005c38-844c-4141-b221-c1c4ebfb4e11" />

@CharlieEriksen can you let me know whether this is fixed? Thanks

@CharlieEriksen is this fixed?

Telemetry should not be an opt out feature, it should be opt in. I really hate finding out another supply chain issue in my stack sending telemetry somewhere.
Repository: confident-ai/deepeval
Issue: Break down LLMTestCase
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Break down LLMTestCase into something more descriptive. In this situation, we will want: 
- RAGTestCase
- ChatbotTestCase
- LLMDataExtractionTestCase
- AgentTestCase

**Describe the solution you'd like**
Add in the above instead of having a generic `LLMTestCase` to provide more robustness is useful. 

**Describe alternatives you've considered**


**Additional context**
Repository: confident-ai/deepeval
Issue: Misuse generate_reason params
Root cause: Validation Gap

When i am running misuse and it is failing because of unknown parameter misuse in the line number 137 in misuse.py file , when I observe the code self.evaluation_template.generate_reason function has no parameter misuse,it is taking misuse_violations as parameter.Can you please fix this bug

Simple fix. Please merge it @penguine-ip
Repository: confident-ai/deepeval
Issue: Deepeval with ragas-metrics supports open-ai only ?
Root cause: Unimplemented Feature Gap

Does Deepeval with ragas-metrics support open-ai only ?

as  I tried  to use to  **gemini-1.5-flash**
EX:

```
  # Replace this with the actual output from your LLM application
    actual_output = "We offer a 30-day full refund at no extra cost."

    # Replace this with the expected output from your RAG generator
    expected_output = "You are eligible for a 30 day full refund at no extra cost."

    # Replace this with the actual retrieved context from your RAG pipeline
    retrieval_context = ["All customers are eligible for a 30 day full refund at no extra cost."]

    metric = RAGASAnswerRelevancyMetric(threshold=0.5,model="gemini-1.5-flash")
    ragas_test_case = LLMTestCase(
        input="What if these shoes don't fit?",
        actual_output=actual_output,
        expected_output=expected_output,
        retrieval_context=retrieval_context
    )
    evaluate(test_cases=[ragas_test_case], metrics=[metric])
```

But iam Getting error : 
```

 TypeError: RAGASAnswerRelevancyMetric.a_measure() got an unexpected keyword argument '_in_component'

..\site-packages\deepeval\metrics\indicator.py:243: TypeError
  f"Invalid model. Available GPT models: {', '.join(model for model in valid_gpt_models)}"
E                   ValueError: Invalid model. Available GPT models: gpt-3.5-turbo, gpt-3.5-turbo-0125, gpt-3.5-turbo-1106, gpt-4-0125-preview, gpt-4-1106-preview, gpt-4-turbo, gpt-4-turbo-2024-04-09, gpt-4-turbo-preview, gpt-4o, gpt-4o-2024-05-13, gpt-4o-2024-08-06, gpt-4o-2024-11-20, gpt-4o-mini, gpt-4o-mini-2024-07-18, gpt-4-32k, gpt-4-32k-0613, gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, gpt-4.5-preview, o1, o1-preview, o1-2024-12-17, o1-preview-2024-09-12, o1-mini, o1-mini-2024-09-12, o3-mini, o3-mini-2025-01-31, o4-mini, gpt-4.5-preview-2025-02-27

..\site-packages\deepeval\models\llms\openai_model.py:
```

@Farahala yup! Please use deepeval's native RAG metrics instead: https://deepeval.com/docs/metrics-answer-relevancy
Repository: confident-ai/deepeval
Issue: Chore: Declare `__all__` in submodules

**Describe the bug**
Currently, most code snippets from the doc use import paths that trigger VSCode's pylance's [warning](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#reportPrivateImportUsage) which is the default for "standard" type checking mode: 

<img width="967" height="105" alt="Image" src="https://github.com/user-attachments/assets/c5472f23-d69f-4261-be85-f8c18cb0df55" />

But even more annoying is that Auto-Import suggestions do not work as well.

**To Reproduce**
e.g. type `GEval` into VSCode, `cmd+.` on it to open quick-fix, no import suggestions will be made.

**Expected behavior**
Each submodule should declare a list of `__all__` that list what should be considered public.

## ✅ Issue Resolved - Pull Request Created

I've successfully fixed the VSCode Pylance import issues and created a pull request:

**�� Pull Request:** #1881- Fix VSCode Pylance Import Issues

### 🔧 What I Fixed
- **Added comprehensive `__all__` declarations** to all key modules
- **Fixed auto-import functionality** for VSCode Pylance (Cmd+. quick-fix now works)
- **Removed import warnings** for standard type checking mode
- **Improved IntelliSense support** across the entire codebase

### 📋 Modules Fixed
- `deepeval/metrics/__init__.py` - All 40+ metrics exported (GEval, AnswerRelevancyMetric, etc.)
- `deepeval/test_case/__init__.py` - All test case classes
- `deepeval/models/__init__.py` - All model classes (GPTModel, AnthropicModel, etc.)
- `deepeval/dataset/__init__.py` - Dataset classes
- `deepeval/evaluate/__init__.py` - Evaluation functions
- `deepeval/tracing/__init__.py` - Tracing functions
- Various submodules (g_eval, answer_relevancy, faithfulness, etc.)

### ✅ Benefits Achieved
- ✅ **No more Pylance warnings** for clean imports
- ✅ **Auto-import suggestions work** (Cmd+. quick-fix now shows suggestions)
- ✅ **Better IntelliSense support**
- ✅ **Cleaner import statements**
- ✅ **Non-breaking change** - all existing code continues to work

### 🧪 Testing
- All imports tested successfully
- Verified with VSCode Pylance
- No breaking changes to existing functionality

### 🎯 Specific Fix for Your Example
The issue you mentioned with `GEval` is now resolved:
- Type `GEval` in VSCode
- Press `Cmd+.` for quick-fix
- **Import suggestions will now appear** 

Please review the PR and let me know if any changes are needed! Peace!!

@JoongWonSeo this is now fixed in the latest release

@divyanshu2003singh i couldn't merge your PR bceause of the extra read me files, was wondering can you check in your vs settings to verify if it works? We don't have pylance, really appreciate it

@penguine-ip I checked in my VSCode setup with Pylance—everything works perfectly. No warnings or errors, and auto-import suggestions are showing up as expected. The fix is confirmed!

Thanks for the fix! Small thing but reduces the development friction quite a bit :)
Repository: confident-ai/deepeval
Issue: Synthesizer.__init__() got an unexpected keyword argument 'embedder'
Root cause: Documentation Deficiency

in documentaion we can use custom embedding to generate golden by making customize Embedding class and then use in in the synthesizer as following 
synthesizer = Synthesizer(model=client,embedder=CustomEmbeddingModel(api_key=""))
but i got TypeError: Synthesizer.__init__() got an unexpected keyword argument 'embedder' and i could find the way to use it 


The docs for the synthesizer don't have a "embedder" argument. Here's the link to the official docs: https://deepeval.com/docs/synthesizer-introduction

Can you share where you've seen the document that shows the "embedder" argument

@A-Vamshi  from also official docs , it is the link ,you can check it  : [https://deepeval.com/guides/guides-using-custom-embedding-models](url)

<img width="1914" height="527" alt="Image" src="https://github.com/user-attachments/assets/2bed3afe-fe93-4cde-bf09-459cc7d948b0" />

Hey @Sara-Hossny that is probably outdated, please see this instead: https://deepeval.com/docs/synthesizer-generate-from-docs#customize-context-construction

Basically use the `embedder` parameter instead.