zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Fix: Improve readability of cards/expandable sections in dark mode

When browsing the [DeepEval Docs](https://deepeval.com/docs/getting-started) in dark mode, all card-style components (e.g., info boxes, expandable sections like “Evaluations getting stuck?”) use light backgrounds. This breaks visual consistency and makes them hard to read against the dark-mode UI.

The issue is not limited to a single section, any "card" or "info box" element appears with a white/grey background, which contrasts poorly with dark mode and reduces readability.

**Steps to Reproduce:**

1. Open [DeepEval Docs Getting Started](https://deepeval.com/docs/getting-started).
2. Enable dark mode (default styling from the site).
3. Observe any info box or expandable card (e.g., “INFO” callout or “Evaluations getting stuck?”).
4. Background stays light, text contrast suffers, making it hard to read in dark mode.

**Expected Behavior:**
Cards/expandable sections should respect dark mode theming, e.g., darker background color, adjusted font color, and consistent styling with the rest of the UI.

**Screenshot:** <img width="982" height="615" alt="Image" src="https://github.com/user-attachments/assets/a285601e-5d9d-4f53-9120-ab379c54cb80" />


Hey, I would like to work on this issue
can i please get assigned ?
Repository: confident-ai/deepeval
Issue: feat: Add per-task timeouts to semaphore guarded awaits in execute.py
Root cause: Unimplemented Feature Gap

### Problem
In `deepeval/evaluate/execute.py`, several coroutines are awaited inside `async with semaphore:` blocks without a timeout. If a downstream call stalls (model, user callback, etc.), the await can block forever and the whole run hangs.

### Goal
Guard semaphore-wrapped awaits with `asyncio.wait_for` using our existing config: `DEEPEVAL_PER_TASK_TIMEOUT_SECONDS`

**What to change**
Find patterns like:

```py
async def execute_with_semaphore(func, *args, **kwargs):
    async with semaphore:
        return await func(*args, **kwargs)
```

Replace the inner `await` with a bounded wait:

```py

async def execute_with_semaphore(func, *args, **kwargs):
    async with semaphore:
        try:
            return await asyncio.wait_for(
                func(*args, **kwargs),
                timeout=settings.DEEPEVAL_PER_TASK_TIMEOUT_SECONDS,
            )
        except asyncio.TimeoutError:
            # Optional: add structured logging here with enough context
            # Then surface a friendly error object / re-raise for caller handling
            # see existing `except asyncio.TimeoutError:` block
            raise
```

If the helper takes a coroutine instead of `func,*args`, use the same `wait_for` around the coroutine:

```py
async def execute_callback_with_semaphore(coro):
    async with semaphore:
        return await asyncio.wait_for(
            coro,
            timeout=settings.DEEPEVAL_PER_TASK_TIMEOUT_SECONDS,
        )
```

### Scope
- File: `deepeval/evaluate/execute.py`
- Only semaphore-guarded awaits that do not already have an enclosing timeout.
- **Don’t** change function signatures or settings names.
- **Don’t** add new settings.

**Testing hints**
Create a dummy coroutine that `await asyncio.sleep(9999)` inside one of the affected paths and verify it times out per setting. For local testing, temporarily set `DEEPEVAL_PER_TASK_TIMEOUT_SECONDS=2` via env or settings to speed up.

**Notes**

* It’s okay if some call sites end up “double guarded” (inner per-task plus outer group timeout), the inner wins for per-task cutoff, outer protects the batch.



Hi! Can you please assign this issue to me? I’d love to work on it.

Hi @trevor-cai 
I’d like to work on this issue. Could you please assign to me?

Thanks!


Hi @yujiiroo and @MRADULTRIPATHI:

I have assigned it to @yujiiroo for about 48 hours. I will re-assign after that if there is no activity. Good luck, and thank you both for offering to contribute.

Thanks for assigning this to me! I’ll get started and aim to open a PR soon.

@yujiiroo remember to add `Fixes #2125` to the description of your PR so that github links your PR to this issue.

alright, i'll open a PR in a few hours.


@trevor-cai can you please review the PR?
Repository: confident-ai/deepeval
Issue: feat: Add per-task timeouts to  `observed_callback` in execute.py
Root cause: Unimplemented Feature Gap

## Guard async `observed_callback` in sync path with a timeout

### Problem
In `deepeval/evaluate/execute.py`, `execute_agentic_test_cases` invokes an async `observed_callback` via `loop.run_until_complete` without a timeout. If the callback stalls due to a model call or user code, it can hang the entire run.

### Goal
Wrap the coroutine with `asyncio.wait_for(` before passing it to `run_until_complete`, using our existing config: `DEEPEVAL_PER_TASK_TIMEOUT_SECONDS`

**What to change**
Find this block `around` lines 831–842:

```py
with Observer(
    "custom",
    func_name="Test Wrapper",
    _progress=progress,
    _pbar_callback_id=pbar_tags_id,
):
    if asyncio.iscoroutinefunction(observed_callback):
        loop = get_or_create_event_loop()
        loop.run_until_complete(observed_callback(golden.input))
    else:
        observed_callback(golden.input)
    current_trace: Trace = current_trace_context.get()
```

Change the async branch to:

```py
from deepeval.config.settings import get_settings
settings = get_settings()

if asyncio.iscoroutinefunction(observed_callback):
    loop = get_or_create_event_loop()
    coro = observed_callback(golden.input)
    loop.run_until_complete(
        asyncio.wait_for(
            coro,
            timeout=settings.DEEPEVAL_PER_TASK_TIMEOUT_SECONDS,
        )
    )
else:
    observed_callback(golden.input)
```

Also in the async path (`_a_execute_agentic_test_case`) around lines 1192–1196, replace:

  ```py
  await observed_callback(golden.input)
  ```

  with:

  ```py
  await asyncio.wait_for(
      observed_callback(golden.input),
      timeout=settings.DEEPEVAL_PER_TASK_TIMEOUT_SECONDS,
  )
```

### Acceptance criteria

- The async `observed_callback` path in `execute_agentic_test_cases` is bounded by `asyncio.wait_for` with `DEEPEVAL_PER_TASK_TIMEOUT_SECONDS`.
- the `observed_callback` in `_a_execute_agentic_test_case` is similarly bounded
- Existing behavior for sync callbacks is unchanged.
- A hung test callback raises `asyncio.TimeoutError` instead of blocking the run.

**Testing hints**

- Create a dummy async callback that does `await asyncio.sleep(9999)`.
- Set `DEEPEVAL_PER_TASK_TIMEOUT_SECONDS=1` (env or test settings).
- Verify the call times out and the rest of the run continues/handles the exception as expected.

 



He @tharun634,

Thank you so much for the PR. May you please drop a comment here so that I can assign you the issue. Github won't let me assign the issue to you unless you have commented on the issue.

Thank you again. I left a comment for you on your PR by the way.

Thanks! Will address the comments shortly
Repository: confident-ai/deepeval
Issue: AmazonBedrockModel fails when using Claude Sonnet 4.5 model due to conflicting temperature and topP parameters
Root cause: Validation Gap

__Describe the bug__

AmazonBedrockModel fails when using Claude Sonnet 4.5 model due to conflicting temperature and topP parameters. The model throws a ValidationException because Claude Sonnet 4.5 cannot accept both `temperature` and `top_p` parameters simultaneously, but AmazonBedrockModel automatically sets both parameters regardless of model compatibility.

__To Reproduce__

Steps to reproduce the behavior:

1. Create an AmazonBedrockModel instance with Claude Sonnet 4.5:

   ```python
   from deepeval.models import AmazonBedrockModel
   bedrock_model = AmazonBedrockModel(
       model_id="anthropic.claude-sonnet-4-5-20250929-v1:0",
       region_name="us-east-1",
       temperature=0
   )
   ```

2. Attempt to generate content using the model

3. Observe ValidationException from Bedrock API

__Expected behavior__ The AmazonBedrockModel should successfully work with Claude Sonnet 4.5 model by only sending compatible parameters. For models that don't support both temperature and top_p simultaneously, only one parameter should be sent.

__Error Message__

```javascript
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the Converse operation: The model returned the following errors: `temperature` and `top_p` cannot both be specified for this model. Please use only one.
```

__Root Cause Analysis__

In `get_converse_request_body()`, the method sets both:

- `temperature`: from the constructor parameter (defaults to 0)
- `topP`: from generation_kwargs with a default of 0

This causes conflicts for models like Claude Sonnet 4.5 that only accept one of these parameters.

__Desktop (please complete the following information):__

- deepeval Version: 3.5.6


Hey @eppeters, this issue was recently solved. We decided to remove the default `temperature` they are now defaulted to what the amazon bedrock model offers, you can set them using the `generation_kwargs` parameter. Here's the docs: https://deepeval.com/integrations/models/amazon-bedrock#python

Let us know if you run into any other issues.

Hi, the prompt management platform has both on by default without the option to remove and this same issue surfaces there.

Hi, the prompt management platform has both on by default without the option to remove and this same issue surfaces there.
Repository: confident-ai/deepeval
Issue: Individual verdicts per test case for Arena Eval
Root cause: Unimplemented Feature Gap

Currently, when we do ArenaGEval, we get winner along with reason.

I'm requesting here, along with above 2 parameter if we can get- with what margin one model is winning, that can really help us decide where my model is bad and where it is too good.

let's say I've 2 models- m1 and m2
m1 can be winner with 51-49 and 89-11 as well. 

For small margin we can neglect it but where the margin is high that can help me with better insights.

I can pick this up.
Repository: confident-ai/deepeval
Issue: `minimum_score` argument of `AnswerRelevancyMetric`'s constructor possibly mistyped as bool
Root cause: Validation Gap

**Describe the bug**
The constructor takes two optional args:
`def __init__(
        self, minimum_score: bool = 0.5, model_type: str = "cross_encoder"
    ):`

`minium_score` typed to bool causes the IDE (PyCharm in my case) to underscore lines like this one
`answer_relevancy_metric = AnswerRelevancyMetric(minimum_score=0.5)`
in grey because `0.5` does not fit `bool` expectation.

**Additional context**
`❯ pip freeze | grep deepeval
deepeval==0.20.8`

Thank you for bringing this up Artur, it will be fixed for the next release (v0.20.12) in 2 day's time.
Repository: confident-ai/deepeval
Issue: UnicodeEncodeError: 'ascii' codec can't encode character in DeepTeam red teaming attacks
Root cause: Validation Gap

**Describe the bug**
I am facing a recurring UnicodeEncodeError when running DeepTeam red teaming attacks against my model callback. Despite configuring my environment fully for UTF-8 (all locale variables, PYTHONIOENCODING, and Python default encoding are UTF-8), and attempting to encode/decode inputs in UTF-16 and UTF-32 inside the callback, the error persists.

**Error :** 
UnicodeEncodeError: 'ascii' codec can't encode character '\u201c' in position 7: ordinal not in range(128)

**To Reproduce**
1. Run the simplest DeepTeam example with a basic bias vulnerability and model_callback returning simple strings.
2. Observe the following error traceback (full traceback included below).
3. Try encoding inputs inside the callback as UTF-16 or UTF-32; error is unchanged.
4. Environment variables:
LANG=en_US.UTF-8
LC_ALL=en_US.UTF-8
PYTHONIOENCODING=utf-8

Python encoding confirmed as UTF-8 by:
`import sys, locale
print(sys.stdout.encoding)
print(sys.getdefaultencoding())
print(locale.getpreferredencoding())
`


**Expected behavior**
DeepTeam should correctly handle Unicode prompts or sanitize content appropriately to avoid passing non-ASCII characters to ASCII-only components (like HTTP headers).

**Screenshots**

<img width="1512" height="858" alt="Image" src="https://github.com/user-attachments/assets/8e39451b-e62d-43d1-a3ff-5fd99ca11727" />

<img width="771" height="646" alt="Image" src="https://github.com/user-attachments/assets/15375c45-bb4b-4b3e-8020-b9dbba8bf37b" />

**Desktop**
 - OS: iOS
 - Browser NA

Name: deepteam
Version: 0.2.6

Name: deepeval
Version: 3.6.6

Python 3.10.18


@mahi2694 please raise this in the deepteam github
Repository: confident-ai/deepeval
Issue: Issue with getting RAGAS metrics
Root cause: External Dependency Breakage

**Describe the bug**
Whenever I try and get the the RAGAS metrics it constantly has an issue with context. even when i use the test case on the DeepEval docs i get the same issue. This is the error: ValueError: Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Sequence'>

Thanks for bringing this up! It's now fixed with the latest release, run `pip install -U deepeval` to continue. 

Is it fixed? I have the same error in all versions that I have tried '0.20.10', '0.20.11', '0.20.12'...

@denyslazarenko Which RAGAS metric are you using? I can take a deeper look

I'm getting this as well when running all ragas metrics but the RagasContextualRelevancyMetric on the latest release.  

Probably unrelated, but for the RagasContextualRelevancyMetric I'm getting a different (context length) error 

```
openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 4097 tokens. However, your messages resulted in 6143 tokens. Please reduce the length of the messages.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}
```

Looking at the RAGAS docs, they say they're using gpt-3.5-turbo so shouldn't be a context length issue.  


@DanLeininger Hey, that would be an issue with openai. Your context length is indeed too long. I think RAGAS allows you to customize what model to use but we haven’t exposed that feature in deepeval, would you mind making a quick PR that does that?
Repository: confident-ai/deepeval
Issue: No Results Table Displayed for MultimodalToolCorrectnessMetric while launching test with assert_test
Root cause: Validation Gap

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Run the following minimal script:

```
from deepeval.metrics import MultimodalToolCorrectnessMetric
from deepeval.test_case import MLLMTestCase, ToolCall
from deepeval import assert_test

question = ["What's in this image?"]
answer = ["The image shows a pair of running shoes."]

tools_called = [ToolCall(name="ToolQuery")]
expected_tools = [ToolCall(name="ImageAnalysis")]


def test_multimodal_tool_correctness_metric():

    test_case = MLLMTestCase(
        input=question,
        actual_output=answer,
        tools_called=tools_called,
        expected_tools=expected_tools
    )

    metric = MultimodalToolCorrectnessMetric(threshold=0.5)
    assert_test(test_case, [metric])
```

**Observed behavior:**
`No test cases found, please try again.`

**Expected behavior**
After test execution, a results table should be displayed.

**Screenshots**

<img width="1378" height="242" alt="Image" src="https://github.com/user-attachments/assets/3b87e1cf-7aea-43aa-b24a-f44e3bc1a30e" />

**Desktop (please complete the following information):**
 - OS: Windows 11
 - Browser N/A
 - Version 3.6.6

**Additional context**
The issue originates in `from deepeval.metrics import BaseMetric` within deepeval/metrics/multimodal_metrics/multimodal_tool_correctness/multimodal_tool_correctness.py

The MultimodalToolCorrectnessMetric inherits BaseMetric instead of BaseMultimodalMetric, and can't work with MLLMTestCase.

As a result, `execute_test_cases()` from deepeval/evaluate/execute.py can't append metric in the mllm_metrics list

```
for metric in metrics:
    metric.async_mode = False
    if isinstance(metric, BaseMetric):
        llm_metrics.append(metric) 
    elif isinstance(metric, BaseConversationalMetric):
        conversational_metrics.append(metric)
    elif isinstance(metric, BaseMultimodalMetric):
        mllm_metrics.append(metric)
```
and skips the test case

```
# No caching and not sending test cases to Confident AI for multimodal metrics yet
elif isinstance(test_case, MLLMTestCase):
    if len(mllm_metrics) == 0:
        continue
```
The test run manager doesn't update

`test_run_manager.update_test_run(api_test_case, test_case)`

The `wrap_up_test_run()` function raises the message `"No test cases found, please try again."`

```
elif (
        len(test_run.test_cases) == 0
        and len(test_run.conversational_test_cases) == 0
    ):
        print("No test cases found, please try again.")
        delete_file_if_exists(self.temp_file_path)
        return
```

Solution

Change the import from BaseMetric to BaseMultimodalMetric in the deepeval/metrics/multimodal_metrics/multimodal_tool_correctness/multimodal_tool_correctness.py.
Change `MultimodalToolCorrectnessMetric(BaseMetric):` to `MultimodalToolCorrectnessMetric(BaseMultimodalMetric ):`

I can try to create a pull request, if needed
Repository: confident-ai/deepeval
Issue: There is no Exception message while running a metric with assert_test and empty metric execution
Root cause: Validation Gap

**Describe the bug**
There is a "1 passed" message after the run of the MultimodalToolCorrectness metric with assert_test and empty metric execution.

**To Reproduce**
Run the following minimal script:

```
from deepeval.metrics import MultimodalToolCorrectnessMetric
from deepeval.test_case import MLLMTestCase, ToolCall
from deepeval import assert_test

question = ["What's in this image?"]
answer = ["The image shows a pair of running shoes."]

tools_called = [ToolCall(name="ToolQuery")]
expected_tools = [ToolCall(name="ImageAnalysis")]


def test_multimodal_tool_correctness_metric():

    test_case = MLLMTestCase(
        input=question,
        actual_output=answer,
        tools_called=tools_called,
        expected_tools=expected_tools
    )

    metric = MultimodalToolCorrectnessMetric(threshold=0.5)
    assert_test(test_case, [metric])
```

**Actual behavior**
There is a `1 passed, 3 warnings in 0.09s` message after the test run, but no metrics were actually executed.

**Expected behavior**
There should be an Exception message about no metric execution, because the metric is incompatible with the test case type.

**Screenshots**

<img width="1378" height="242" alt="Image" src="https://github.com/user-attachments/assets/bcba5377-a2ae-420d-b3bc-6b221095fe81" />

**Desktop (please complete the following information):**
 - OS: Windows 11
 - Browser N/A
 - Version 3.6.6

**Additional context**
The issue originates in the `assert_test()` function within deepeval/evaluate/evaluate.py

There are no Exception messages for the case when you run the metric with an empty metric execution.
In this case:
1. `MultimodalToolCorrectnessMetric` incorrectly inherits from `BaseMetric` instead of `BaseMultimodalMetric`
2. Due to the type mismatch, the metric is not executed with `MLLMTestCase`
3. The test passes with `1 passed` even though no metrics were run.
4. `assert_test()` in `deepeval/evaluate/evaluate.py` doesn't validate that metrics were actually executed

Possible solution

Add a check for the case when you run the metric with an empty metric execution in the `assert_test()` function (after the 160th line).

```
        _use_bar_indicator=False,
        _is_assert_test=True,
    )[0]
if len(test_result.metrics_data) == 0:
    error_msg = [
        "No metrics were executed for the test case.",
    ]
raise AssertionError("\n".join(error_msg))
```
Related issue

[https://github.com/confident-ai/deepeval/issues/2182]

Hey @EZLOGIC, thanks for the issue, we fixed this in a recent PR. Gonna close this and the previous issue.
Repository: confident-ai/deepeval
Issue: dataset save to json  format_turns json.dumps ensure_ascii set false
Root cause: Configuration Error

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
When saving the dataset in JSON format, the data in "turn" should not be converted to ASCII format.

change deepeval/dataset/utils.py format_turns function as follows:
```python
def format_turns(turns: List[Turn]) -> str:
    res = []
    for turn in turns:
        cur_turn = {
            "role": turn.role,
            "content": turn.content,
            "retrieval_context": (
                turn.retrieval_context if turn.retrieval_context else None
            ),
        }
        res.append(cur_turn)
    try:
        return json.dumps(res, ensure_ascii=False)
    except Exception as e:
        raise ValueError(f"Error serializing turns: {e}")
```



Hey, thanks for the PR on this one, I'm gonna close the issue now.
Repository: confident-ai/deepeval
Issue: Factual consistency metric calculation is incorrect
Root cause: Algorithmic Error

**Describe the bug**
In factual_consistency.py the factual consistency metric calculation is incorrect (see lines 55-58). This is how I'm reading your code: Context is chunked into a list of strings; then each chunk is compared to the output; whichever chunk has the highest score -- that score becomes the "overall" evaluation score. If that's true, then you compare only one piece (chunk) of the context to the entire output. However, evaluation should take entire context into account. If your entire context says "flowers can be deadly and poisonous", the chunk (n=20 characters) "flowers can be" will likely have a high comparison (in your case"evaluation") score to the output "flower can be beautiful and fragrant." In which case evaluation score will indicate an inaccurate relationship between your output and context.

Hope this helps.


It certainly does Madina! It will be fixed in our next release (v0.20.12), really appreciate it!

@MadinaKamolova It's now deprecated for now, added a new hallucination metric as a replacement: https://docs.confident-ai.com/docs/evaluation-metrics#hallucination, come join our discord! https://discord.com/invite/a3K9c8GRGt
Repository: confident-ai/deepeval
Issue: No arguments to create embedding model instance
Root cause: Architectural Constraint

I would like to create an instance of an embedding model and pass it into a funtion, but the constructor only retrieves env variables, it does not accept any arguments?

https://github.com/confident-ai/deepeval/blob/fe2e8d16252d0a559fb4b72a13a4e295442d234a/deepeval/models/embedding_models/azure_embedding_model.py#L21

The regular model however does this:

https://github.com/confident-ai/deepeval/blob/main/deepeval/models/llms/azure_model.py#L28

Instead of passing it as an argument, you can set your azure model as follows:
```
deepeval set-azure-openai \
    # e.g. https://example-resource.azure.openai.com/
    --openai-endpoint=<endpoint> \
    --openai-api-key=<api_key> \
    # e.g. gpt-4.1
    --openai-model-name=<model_name> \
    # e.g. Test Deployment
    --deployment-name=<deployment_name> \
    # e.g. 2025-01-01-preview
    --openai-api-version=<openai_api_version> \
    --model-version=<model_version> # e.g. 2024-11-20
```

and run the following

```
deepeval set-azure-openai-embedding --embedding-deployment-name=<embedding_deployment_name>
```

Here are the docs btw: https://deepeval.com/guides/guides-using-custom-embedding-models#using-azure-openai

Lemme know if this isn't working for you or it's not what you're looking for

I know, but I'm creating a dockerized container to ship, and I don't want to have a setup script. I also use the returned models for my scripts since many of them use different models for different tests. My current implementation have to take in the parameters as args and set the env variables before then initializing the embedding model and passing it on. It can still use the env variables if no arguments are given, I just find it weird that arguments can't be given, and my PR in mye company was declined because of bad coding standards.

@SindreMHegre i believe its solved and will be released very soon in latest release :)