zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Don't push for confidently.ai platform, or give way to silence it
Root cause: Architectural Constraint

**Describe the bug**
Running evals without doing a `deepeval login` results a console message being printed. 
https://github.com/confident-ai/deepeval/blob/main/deepeval/test_run/test_run.py#L744

I would like to make sure this library's stance is to be less opinionated about cloud platforms. Those of us at large companies should be not be shipping data outside our compute boundaries. 

**To Reproduce**
Steps to reproduce the behavior:
1. Run any eval without logging in

**Expected behavior**
If toggled, no references to confidently.ai, respecting local contexts better


The cofounder @penguine-ip  closed a similar issue, #1633 🤷  I guess I'll switch to something else

I agree, this message keeps burning tokens with our coding assistants which run these tests. 
Everybody is aware of the cloud platform, no need to spam and annoy us.
Repository: confident-ai/deepeval
Issue: False login when using login_with_confident_api_key with no key
Root cause: Validation Gap

If I try to login within a python script via `deepeval.login_with_confident_api_key()` I'll get a login success message even if I don't provide a key.

How to reproduce:
Run this:
```python
import deepeval

deepeval.login_with_confident_api_key(None)
```

You should get this response which is false:
```
🎉🥳 Congratulations! You've successfully logged in! 🙌
```

Expected behavior:
It should fail/ask for a key.
Repository: confident-ai/deepeval
Issue: Auto-create evaluation dataset and edge cases
Root cause: Unimplemented Feature Gap

Develop an automated way to automatically create an evaluation dataset with edge cases so that users don't have to write tests. Then make it super easy to run! 

New Design Plan: 
- [ ] Prompt to generate tests - ensure to include edge cases and RAG performance
- [ ] Function to run evaluation on the generated tests
- [ ] Improve the functionality
Repository: confident-ai/deepeval
Issue: Fix SQuAD Extractive Answer Prediction by Using StringSchema Instead of MultipleChoiceSchemaLower
Root cause: Architectural Constraint

### Bug Description
When running the SQuAD benchmark, the model’s prediction is consistently returning `"a"` instead of the expected extractive answer (e.g., `"solar"`, `"Rankine"`, etc.). This happens because the benchmark uses the `MultipleChoiceSchemaLower`, which is meant for multiple-choice questions—instead of a schema that expects a full text answer directly extracted from the context.

### To Reproduce
1. Configure the SQuAD benchmark with a task (e.g., Steam Engine) and run it:
```python
benchmark = SQuAD(
    tasks=[SQuADTask.STEAM_ENGINE],
    n_shots=5,
    evaluation_model="o3-mini",
    n_problems_per_task=5,
)
benchmark.evaluate(model=custom_llm)
```

2. Observe that for every example, the prediction returned is `"a"` and the score is 0.

### Proposed Solution  
Replace every usage of `MultipleChoiceSchemaLower` with `StringSchema` in the SQuAD benchmark.

1. **Update the Import Statement:**  
   Change the import from:
   ```python
   from deepeval.benchmarks.schema import MultipleChoiceSchemaLower
   ```
   to:
   ```python
   from deepeval.benchmarks.schema import StringSchema
   ```

2. **Update All Usages:**  
   Ensure that every part of the SQuAD benchmark class that currently uses `MultipleChoiceSchemaLower` is modified to use `StringSchema`. This change ensures that the model's output is processed as a full text extractive answer, rather than as a multiple-choice option.
Repository: confident-ai/deepeval
Issue: Spellling error in prompt

The spelling error may be intentional or just the consequence of using an actual example. Still, if intentional a note indicating a known error wouldn't hurt.

The follow guide includes prompt test mispelling the word how (spelt as gow). 

from deepeval.test_case import LLMTestCase
...

test_case = LLMTestCase(
    input="I'm on an F-1 visa, **gow** long can I stay in the US after graduation?",
    actual_output="You can stay up to 30 days after completing your degree.",
    expected_output="You can stay up to 60 days after completing your degree.",
    retrieval_context=[
        """If you are in the U.S. on an F-1 visa, you are allowed to stay for 60 days after completing
        your degree, unless you have applied for and been approved to participate in OPT."""
    ]
)

https://github.com/confident-ai/deepeval/blob/main/docs/guides/guides-rag-evaluation.mdx
Repository: confident-ai/deepeval
Issue: Deepeval's Python version compatibility cause Poetry version issues
Root cause: Configuration Error

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
I noticed that DeepEval has not updated in my project for a very long time. I researched and discovered why.

DeepEval's Python version requirements. I'm not sure why other DeepEval users are not having this problem, but its easy for me to reproduce.


**To Reproduce**
Steps to reproduce the behavior:
```
❯ poetry init --name=Test --description="" --author="foo@bar.com" --dependency=deepeval --quiet
❯ cat pyproject.toml                                                                           
[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["..."]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"
deepeval = "^2.5.5"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

❯ poetry lock
Updating dependencies
Resolving dependencies... (0.2s)

The current project's supported Python range (>=3.10,<4.0) is not compatible with some of the required packages Python requirement:
  - deepeval requires Python <3.14,>=3.9, so it will not be satisfied for Python >=3.14,<4.0

Because no versions of deepeval match >2.5.5,<3.0.0
 and deepeval (2.5.5) requires Python <3.14,>=3.9, deepeval is forbidden.
So, because test depends on deepeval (^2.5.5), version solving failed.

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties
    
    For deepeval, a possible solution would be to set the `python` property to ">=3.10,<3.14"

    https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies,
    https://python-poetry.org/docs/dependency-specification/#using-environment-markers
```

**Expected behavior**
There should be no error message.

Deepeval should work with every future Python version UNLESS it is known there is a problem. In which case the problem should be fixed in deepeval and not with version locks.

**Screenshots**

<img width="931" alt="Image" src="https://github.com/user-attachments/assets/3660f663-bf8d-4c94-a550-2c988317fd4a" />

**Desktop (please complete the following information):**
 - OS: MacOS


@prescod thanks for bringing it up its fixed in the latest deepeval version
Repository: confident-ai/deepeval
Issue: prettify_list() Converts Non-ASCII Characters to Unicode Escape Sequences

**❗BEFORE YOU BEGIN❗**  
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt  

### **Describe the bug**  
When calling `prettify_list()` on a list containing Pydantic models with non-ASCII characters (such as Chinese), the function returns a JSON string with Unicode escape sequences instead of properly displaying the characters.  

This happens because `json.dumps(jsonObj, indent=4)` defaults to `ensure_ascii=True`, which converts non-ASCII characters into Unicode escape sequences (e.g., `"你好"` becomes `"\u4f60\u597d"`).  

### **To Reproduce**  
Steps to reproduce the behavior:  
1. Create a Pydantic model with a field containing non-ASCII characters (e.g., Chinese).  
2. Call `prettify_list()` on a list containing an instance of the model.  
3. The output will display Unicode escape sequences instead of the actual characters.  

Example code to reproduce the issue:  
```python
from pydantic import BaseModel
from deepeval.utils import prettify_list

class ExampleModel(BaseModel):
    text: str

example_instance = ExampleModel(text="你好")
print(prettify_list([example_instance]))
```

### **Expected behavior**  
The function should output properly formatted JSON without Unicode escape sequences when dealing with non-ASCII characters.  

For example, instead of:  
```json
[
    {
        "text": "\u4f60\u597d"
    }
]
```
It should output:
```json
[
    {
        "text": "你好"
    }
]
```

### **Suggested Fix**  
Modify the function in `deepeval/utils.py` by adding `ensure_ascii=False` to `json.dumps`:
```python
formatted_elements.append(
    json.dumps(jsonObj, indent=4, ensure_ascii=False).replace("\n", "\n    ")
)
```
This ensures that non-ASCII characters are displayed correctly in the formatted JSON output.

### **Desktop:**  
- OS: Windows
- Python Version: 3.10.14
- deepeval: 2.5.5


@milk333445 i believe it is resolved :)
Repository: confident-ai/deepeval
Issue: Using OpenAI on Azure fails with TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str' exception
Root cause: Interface Contract Mismatch

deepeval v2.5.5

Getting this exception when calling evaluate on geval metric with AzureOpenAILLM object:
 File ".../lib/python3.9/site-packages/deepeval/metrics/g_eval/g_eval.py", line 271, in _a_evaluate
    self.evaluation_cost += cost
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'

I'm trying to evaluate metrics using openAI azure model like this:

    client = openai.AzureOpenAI(
        api_key=azure_openai_key,
        base_url=azure_openai_endpoint,
        api_version=azure_api_version 
    )

    class AzureOpenAILLM(DeepEvalBaseLLM):
    #     """Custom wrapper for Azure OpenAI to use with DeepEval"""
        def load_model(self):
            """No explicit loading required for Azure OpenAI."""
            pass  # Azure models are accessed via API calls
    #
        def get_model_name(self):
            """Return the model name (deployment name in Azure)."""
            return azure_deployment_name
    #
        def generate(self, prompt: str) -> str:
            """Sends a request to Azure OpenAI and returns only the response text."""
            response = client.chat.completions.create(
                model=azure_deployment_name,  # Use deployment name, not model name
                messages=[{"role": "system", "content": "You are a helpful AI assistant."},
                          {"role": "user", "content": prompt}],
                temperature=0.7,
            )
            return response.choices[0].message.content  # Extract response text
    #
        async def a_generate(self, prompt: str) -> str:
            """Async version of generate."""
            return self.generate(prompt)
    #
        async def a_generate_raw_response(self, prompt: str, **kwargs):
            """Handles unexpected arguments to avoid errors."""
            response = client.chat.completions.create(
                model=azure_deployment_name,
                messages=[{"role": "system", "content": "You are a helpful AI assistant."},
                          {"role": "user", "content": prompt}],
                temperature=0.7,
            )
            text = response.choices[0].message.content
            return response, text  # Return raw response and extracted text
Repository: confident-ai/deepeval
Issue: programmatic alternative to KEY_FILE_HANDLER.write_key
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

I notice people using KEY_FILE_HANDLER.write_key directly to write .deepeval inside of programs to assign config.

**Describe the solution you'd like**

Inside a program, a programmatic, in-memory way would be more ideal, like backed by a dict.

**Describe alternatives you've considered**

just write a file like this and delete it again, but it is less ideal:
```python
    os.remove(KEY_FILE)
    KEY_FILE_HANDLER.write_key(KeyValues.LOCAL_MODEL_NAME, eval_model)
    KEY_FILE_HANDLER.write_key(KeyValues.LOCAL_MODEL_BASE_URL, openai_base_url)
    KEY_FILE_HANDLER.write_key(KeyValues.USE_LOCAL_MODEL, "YES")
    KEY_FILE_HANDLER.write_key(KeyValues.USE_AZURE_OPENAI, "NO")
    KEY_FILE_HANDLER.write_key(KeyValues.LOCAL_MODEL_FORMAT, "json")
```

**Additional context**
Repository: confident-ai/deepeval
Issue: Remove warnings 

```
========================================================= warnings summary =========================================================
../../../../../opt/homebrew/lib/python3.11/site-packages/_pytest/config/__init__.py:1204
  /opt/homebrew/lib/python3.11/site-packages/_pytest/config/__init__.py:1204: PytestAssertRewriteWarning: Module already imported so
cannot be rewritten: deepeval
    self._mark_plugins_for_rewrite(hook)

../../../../../opt/homebrew/lib/python3.11/site-packages/pkg_resources/__init__.py:121
  /opt/homebrew/lib/python3.11/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an 
API
    warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)

../../../../../opt/homebrew/lib/python3.11/site-packages/pkg_resources/__init__.py:2870
  /opt/homebrew/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to 
`pkg_resources.declare_namespace('google')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See 
https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)
    ```

#145
Repository: confident-ai/deepeval
Issue: Deepeval & Pytest version
Root cause: External Dependency Breakage

Hi team,

I'm encountering a dependency resolution issue where our project requires pytest 8.3.5, but deepeval (v2.6.0) restricts pytest to versions below 8.0.0. We prefer not to downgrade our pytest version. Could you clarify the challenges or issues that led to this strict version constraint? I'm attaching the output from:

```
curl -s "https://pypi.org/pypi/deepeval/2.6.0/json" | jq '.info.requires_dist'
```


```js
[
  "requests<3.0.0,>=2.31.0",
  "tqdm<5.0.0,>=4.66.1",
  "pytest<8.0.0,>=7.4.3",
  "pytest-xdist",
  "pytest-repeat",
  "pytest-rerunfailures<13.0,>=12.0",
  "pytest-asyncio<0.22.0,>=0.21.1",
  "tabulate<0.10.0,>=0.9.0",
  "sentry-sdk<2.0.0,>=1.33.1",
  "rich<14.0.0,>=13.6.0",
  "coverage",
  "black",
  "portalocker",
  "openai",
  "twine==5.1.1",
  "aiohttp",
  "typer",
  "instructor",
  "ollama",
  "setuptools",
  "wheel",
  "langchain_openai",
  "langchain_community",
  "llama-index",
  "tenacity<=9.0.0",
  "opentelemetry-api<2.0.0,>=1.24.0",
  "opentelemetry-sdk<2.0.0,>=1.24.0",
  "opentelemetry-exporter-otlp-proto-grpc<2.0.0,>=1.24.0",
  "grpcio<2.0.0,>=1.67.1"
]
```

This output confirms the current dependency constraint on pytest. I appreciate any insight or plans for broadening the supported versions.

Best regards,

I have a similar issue but related the sentry-sdk version being constrained to <2.0.0

@sergeyklay @Tsdevendra1 thanks for bringing this up i removed it in the latest version 2.6.2

Thank you for the swift response and fix. I've just tested version 2.6.2 and can confirm that the issue is resolved. I really appreciate your quick turnaround!


No problem @sergeyklay , will get to your PR soon, thank you in advance!
Repository: confident-ai/deepeval
Issue: Provide ability to disable tqdm
Root cause: Unimplemented Feature Gap

Hi team,

We're experiencing an issue with the current progress bar in deepeval's synthesizer - it tends to clutter our logs. Since there's no native option to disable tqdm, we've implemented a workaround by overriding its initializer:

```py
from functools import partialmethod

from deepeval.synthesizer import Synthesizer
from tqdm import tqdm


tqdm.__init__ = partialmethod(tqdm.__init__, disable=True)

synthesizer = Synthesizer()
synthesizer.generate_goldens_from_contexts(
    # ...
)

```

While this approach works, it feels a bit hacky. It would be great if you could consider adding a built-in option (e.g., a `show_progress_bar` parameter) to disable the progress bar in a more canonical and obvious way.

Thanks for your consideration.

---

P.S. Also it would be nice to have an ability to disable this too:
```py
print(f"Synthetic goldens saved at {full_file_path}!")
```

I am encountering the same issue while using the Synthesizer too. I tried using the method you mentioned, but this disables the pbar globally, which is not ideal for my use case (I want to see the progress bar update for the overall process, but not display the pbar for the interior golden generation). 

I located the `progress_context.py` which is referenced in `synthesizer.py`. The original code block is: 

```
if not progress_bar
            if async_mode:
                with async_tqdm_bar(
                    total=max_generations, desc=description, file=sys.stderr
                ) as progress_bar:
                    yield progress_bar
            else:
                with tqdm_bar(
                    total=max_generations, desc=description, file=sys.stderr
                ) as progress_bar:
                    yield progress_bar
        else:
            yield progress_bar
```

You can modify it to `if progress_bar is not None` to address this temporarily. Hope this helps :)
Repository: confident-ai/deepeval
Issue: Deepval pytest-asyncio dependency version
Root cause: External Dependency Breakage

Hey team,

I’ve encountered a dependency issue when upgrading to Deepval’s latest version, `2.6.2`. Deepval restricts `pytest-asyncio` to `<0.22.0`, but we’re currently running version `0.25.3`. We’d prefer not to downgrade `pytest-asyncio` since we aim to keep our packages up to date.

Any chance of upgrading it to 0.25.3.

Thanks 

hey @said-ali its fixed in the latest 2.6.3 version, can you see if it works? Thanks!

Thank you
Repository: confident-ai/deepeval
Issue: TestResult misses additional_metadata
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I want to group my test results by some attributes  (e.g. complexity - low, medium, high) that i declared in my dataset to see the performance of my model in different areas.

Unfortunately, TestResult objects does not have the additional_metadata field, so I need to do the mapping from LLMTestCase to TestResult myself.

**Describe the solution you'd like**
When running .evaluate on my EvaluationDataset object, I want the additional metadata to be filled from LLMTestCase to TestResult (same as the mapping from expected output, or context for example, which is included in the TestResult object)

**Describe alternatives you've considered**
I could also create datasets for each attribute myself, though then the number of sets is not determined by the number of usecases, but the number of usecases times the number of groups/attributes.

**Additional context**
- from:
```python
@dataclass
class TestResult:
    """Returned from run_test"""

    name: str
    success: bool
    metrics_data: Union[List[MetricData], None]
    conversational: bool
    multimodal: Optional[bool] = None
    input: Union[Optional[str], List[Union[str, MLLMImage]]] = None
    actual_output: Union[Optional[str], List[Union[str, MLLMImage]]] = None
    expected_output: Optional[str] = None
    context: Optional[List[str]] = None
    retrieval_context: Optional[List[str]] = None
```


- to
```python
@dataclass
class TestResult:
    """Returned from run_test"""
    ...
    additional_metadata: dict = None # THIS IS NEW
```
Repository: confident-ai/deepeval
Issue: Synthesizer is not able to take default model as a critic model
Root cause: Architectural Constraint

As a user I expect synthesizer to utilize the default model that was passed during the initialization. This is not the case right now because the `ContextConstructionConfig` is forcing me to pass model in the argument. 

The current initialization looks like this: 
```python
synthesizer = Synthesizer(
    model=azure_openai_llm,
    embedder=azure_embedding_model
)

synthesizer.generate_goldens_from_docs(
    document_paths=['example.txt'],
    context_construction_config=ContextConstructionConfig(
        chunk_size=100,
        chunk_overlap=20,
    )
)
```
In the above code the user is bound to pass custom llm 2 times. 

This design is more sophisticated (imo):
```python
synthesizer = Synthesizer(
    model=azure_openai_llm,
    embedder=CustomEmbeddingModel(),
)

synthesizer.generate_goldens_from_docs(
    document_paths=['example.txt'],
    context_construction_config=ContextConstructionConfig(
        chunk_size=100,
        chunk_overlap=20,
    )
)
```
Repository: confident-ai/deepeval
Issue: Custom Nodes for DAG
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I was trying to implement a DAG metric and needed to create nodes with simple logic (parsing ints out of expected output and actual output strings and then finding the difference of two int's for example) but couldn't because the only options for nodes are LLM based.

**Describe the solution you'd like**
I want to be able to create Custom Nodes within the DAG metric, so I can apply my own logic when needed, while still having the option and support for LLM based nodes.

**Describe alternatives you've considered**
Not many alternatives here, just want to be able to have my own logic within the DAG when I need.

**Additional context**
No additional context


Hey @cchetes sounds like a good feature to build. Could you please elaborate a little bit more if possible? Either a code snippet or in words. 

Seconding this.

Here is an example practical use-case I currently would like to implement: XNOR-gate.

Process: Im trying to determine if an actual_output given, though not equal to the expected_output, may be a correct _alternative_ answer to the given input. While this is (at least for my use-case) hard to evaluate, one sanity check is that the two outputs don't entirely contradict each other on general feasibility..
 
node_1: "Does expected_output indicate that there is no (correct) answer to the given input based on the existing context?"
node_2: "Does actual_output ....."

XNOR-gate: node_1._verdict == node_2._verdict

There are certainly plenty more use-cases for similar logical comparisons etc..

@georg-wind @cchetes Were you able to do this?

I want to understand if there is anything that limiting [BaseNode](https://github.com/confident-ai/deepeval/blob/2e2886e34be941f249b68702b53ca7ca82eff15f/deepeval/metrics/dag/nodes.py#L26) that makes it not extendable?

I'm new to DeepEval as well. Finding my way around
Repository: confident-ai/deepeval
Issue: OllamaEmbeddingModel returns 404 when attempting to embed texts
Root cause: Interface Contract Mismatch

It appears that the `OllamaEmbeddingModel` is using the `OpenAI` client to communicate with an ollama instance. This will end up throwing 404s as the OpenAI URI structure is different than ollama's.

When `a_embed_text` is called, a POST gets sent to `<base_url>/embeddings`, while ollama expects these at `<base_url>/api/embed`.

https://github.com/confident-ai/deepeval/blob/1f47e88d6afeda5340dde6b0c385e42b428d2a51/deepeval/models/embedding_models/ollama_embedding_model.py#L24

Since deepeval is already using some langchain_x packages, we could probably just add `langchain_ollama` and use `OllamaEmbeddings` in load_model.

It appears that Ollama does support the OpenAI request format, you just have to set your `base_url` to include `/v1` at the end so requests go to `<ollama-host>/v1/embedding` instead. This is a bit confusing as `OllamaModel` uses the `ollama` base library which works with a bare base_url.

Resolved by #1474