zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: CybersecurityGuard.name property returns None causing Pydantic validation error
Root cause: Validation Gap

**Describe the bug**
The `__name__` property in the CybersecurityGuard class is not returning a value, causing a Pydantic validation error when using `guard_input()`. The property is defined with a docstring but no return statement, which results in None being returned instead of the guard name.

**To Reproduce**
Steps to reproduce the behavior:
1. Install deepeval version 2.3.0
2. Create a new Python script with the following code:
```python
from deepeval.guardrails import CybersecurityGuard, GuardType, Guardrails
   
cyber_guard = CybersecurityGuard(
      guard_type=GuardType.INPUT,
      purpose="security"
 )
guardrails = Guardrails(
      guards=[cyber_guard]
 )
message = "Is the earth flat"
guard_result = guardrails.guard_input(input=message)
```

3. Run the script
4. Observe the following error: 
`pydantic_core._pydantic_core.ValidationError: 1 validation error for ApiGuard
guard
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]`

**Expected behavior**
The `__name__` property in CybersecurityGuard should return "Cybersecurity Guard" as a string, allowing the `guard_input()` function to properly validate the input without raising a Pydantic validation error.

**Screenshots**
![Image](https://github.com/user-attachments/assets/ea8b4ab2-b1df-4821-bdd0-9ba44c938b45)

**Desktop (please complete the following information):**
 - OS: Windows
- Python Version: 3.12
- deepeval Version: 2.3.0

**Smartphone (please complete the following information):**
N/A

**Additional context**
N/A
Repository: confident-ai/deepeval
Issue: query about trying to add test cases from csv file
Root cause: Documentation Deficiency

I want to add test cases from csv file and relevant chunks is list saved in csv how to read it correctly : 
dataset.add_test_cases_from_csv_file(
    # file_path is the absolute path to you .csv file
    file_path="/kaggle/input/data_English.csv",
    input_col_name="question",
    actual_output_col_name="model_output",
    expected_output_col_name="reference_answer",
    retrieval_context_col_name="relevant_contexts",# is list 
)

@Sara-Hossny what's your question?

it read it as one str not list for example ["[context1,context2,context3]"]
expected  [context1,context2,context3]

On Mon, 3 Feb 2025 at 09:42, Jeffrey Ip ***@***.***> wrote:

> @Sara-Hossny <https://github.com/Sara-Hossny> what's your question?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/confident-ai/deepeval/issues/1333#issuecomment-2630187942>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AYZVBBSS4OCSMAPEB75N2ML2N4MWDAVCNFSM6AAAAABWKJAGDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMZQGE4DOOJUGI>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


@Sara-Hossny did you provide a delimiter in the method to load test cases?

Delimiter is not vaild in case it is aready stored as list as you couldn't
find common delimiter

On Tue, Feb 11, 2025, 5:39 PM Jeffrey Ip ***@***.***> wrote:

> @Sara-Hossny <https://github.com/Sara-Hossny> did you provide a delimiter
> in the method to load test cases?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/confident-ai/deepeval/issues/1333#issuecomment-2651196396>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AYZVBBQEPWWHHPPHIKEB2MD2PIKSHAVCNFSM6AAAAABWKJAGDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJRGE4TMMZZGY>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


@Sara-Hossny how do you store a list in a csv without a delimiter?
Repository: confident-ai/deepeval
Issue: AttributeError: 'NoneType' object has no attribute 'score'
Root cause: Validation Gap

**❗BEFORE YOU BEGIN❗**

**Describe the bug**

![Image](https://github.com/user-attachments/assets/e26238b1-b078-4057-987b-f833793e231d)

**To Reproduce**
1- first i have test cases using dataset.add_test_cases_from_csv_file 
then i use one of them to calculate the relevance using gpt4

**Expected behavior**
get the score of relevance


**Desktop (please complete the following information):**
 - OS: windows




this is content of test case 

![Image](https://github.com/user-attachments/assets/c4cdbbc7-4f97-4f03-a0b5-bf91a7e33455)

it have solved be call measure first seperate then score 
answer_relevancy_metric.measure(dataset.test_cases[0])
answer_relevancy_metric.score
Repository: confident-ai/deepeval
Issue: Creating Custom EvaluationDataset test_case but can't use 'Evaluate' methods
Root cause: Validation Gap

Hi,
First of all, I created a function with pd.apply for the test case, but every time I was getting the error
Event loop is already running. Applying nest_asyncio patch to allow async execution...
Instead, I discovered DataEvulation in the documentation, but it caused the problem below;

There is an my example creating custom Test set with EvaluationDataset and trying to evaluate any tests, I saw true number of indexes in csv but raise a some error like ;

**ValidationError: 1 validation error for LLMApiTestCase
  Value error, Either 'input' and 'actualOutput' or 'multimodalInput' and 'multimodalActualOutput' must be provided. [type=value_error, input_value={'name': 'test_case_0', '...onal_instance_id': None}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.10/v/value_error**

my code bellow;

```
from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()

# Add as test cases
dataset.add_test_cases_from_csv_file(
    # file_path is the absolute path to you .csv file
    file_path="random_path.csv",
    input_col_name="prompt",
    actual_output_col_name="actual_output",
    expected_output_col_name="expected_output"
    #context_col_name="context",
    #context_col_delimiter= ";",
    #retrieval_context_col_name="retrieval_context",
    #retrieval_context_col_delimiter= ";"
)

```

And creating a metric

```
correctness_metric = GEval(
    name="Correctness",
    evaluation_steps=[
        "Check whether the facts in 'actual output' contradicts any facts in 'expected output'",
        "You should also heavily penalize omission of detail",
        "Vague language, or contradicting OPINIONS, are OK",
        
    ],
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
)

evaluate(dataset, metrics=[correctness_metric])
```
Edit: 
In dataset I saw input's are None type but in csv file it isn't

Found! if you guys trying so much times a data evaluation methods your input can be collapse 😆 
If you take this code it'll be working so well, but careful if you chose openai api u'll be wait so much seconds for open server responses
Have a good day ✋
Repository: confident-ai/deepeval
Issue: New Relic License key and endpoint are exposed in source

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

No.

**Describe the bug**
The New Relic license key and endpoint are exposed directly in source code and could be used maliciously. This is true in the current version and many historical versions.
https://github.com/confident-ai/deepeval/blob/v2.2.2/deepeval/telemetry.py#L76
https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/

**To Reproduce**
It's visible directly in source.
https://github.com/confident-ai/deepeval/blob/v2.2.2/deepeval/telemetry.py#L76


**Expected behavior**
https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/
It seems like there are some potential security concerns with the New Relic license keys being exposed, so this key being exposed should be reviewed to make sure it's not over-permissioned and that this usage is ok.


**Additional context**
Flagged by an automated tool as disclosed secrets in source, so I'm making a ticket.

Hey @paul-abb we'll probably move it once someone starts attacking it (i probably shouldn't say it in a public forum too)
Repository: confident-ai/deepeval
Issue: Change to a version of gRPC >= 1.70.0 as the current verison (1.67.1) has a known vulnerability.
Root cause: External Dependency Breakage

**Describe the bug**  
[[Blackduck](https://www.blackduck.com/)](https://www.blackduck.com/) (an application security testing and software composition analysis tool) reports that gRPC contains a data corruption flaw in its zero-copy transmission functionality. This flaw could be exploited by a remote attacker to cause a denial-of-service (DoS). The `opentelemetry-exporter-otlp-proto-grpc` library uses `gRPC` with a flexible dependency: `grpcio >= 1.63.2, < 2.0.0`. Updating to a newer version (e.g., 1.70.0) of `gRPC` would resolve the vulnerability.

**To Reproduce**  
Steps to reproduce the behavior:
1. Perform an analysis using Blackduck.

**Expected behavior**  
The Blackduck analysis should report no vulnerabilities related to gRPC.

**Screenshots**  
<img width="1356" alt="Image" src="https://github.com/user-attachments/assets/596171a7-0c42-402b-84da-3acc9b839d2a" />

**Environment:**  
 - OS: macOS Sequoia v15.3  
 - Browser: Chrome  
 - Application Version: Deepeval 2.2.7  
 - Python: 3.11  

**Additional context**  
A similar PR was made previously: https://github.com/confident-ai/deepeval/pull/1228
Repository: confident-ai/deepeval
Issue: Feature Request: Return both score and reason from metric.measure()
Root cause: Unimplemented Feature Gap

## Description
Currently, metric.measure() only returns the score, while evaluate() returns both score and reason. Since measure() internally calls evaluate() and stores the reason in self.reason, it would be more convenient if measure() returned both values directly.

This would be particularly useful when evaluating in batches, as it avoids the need to call metric.reason separately for each instance. Instead of handling post-processing in multiple steps, it would be cleaner and more efficient to get both score and reason as a return value from measure().

## Proposed Solution
Modify metric.measure() to return a tuple (score, reason), similar to what evaluate() provides, rather than requiring users to retrieve metric.reason separately.

This would make the API more intuitive and reduce extra steps in evaluation pipelines.
Would love to hear your thoughts on this.

Hey @Mujae thanks for the suggestion! The think with reason is it is not required if `include_reason` is False. For those that need both score and reason they can just do `metric.score` and `metric.reason`.

Thank you for your response! 
Yes, that's correct. However, in the case of G-eval, it always returns a reason, so there is no include_reason option. Given that, if measure itself returns the reason, there would be no need to access metric.reason separately, which is why I made the suggestion.
Repository: confident-ai/deepeval
Issue: o3-mini is not working with G-Eval due to wrong temperature and logprobs parameter
Root cause: Configuration Error


**Describe the bug**
Using a custom G-Eval metric with o3-mini does not work at moment because the OpenAI API is called with temperature = 0.7 and logprobs = True.

**To Reproduce**
Using a custom G-Eval metric with o3-mini and try to measure it. The API call to OpenAI will fail.

**Expected behavior**
The temperature should be 1.0 and logprobs = False (or omitted entirely).

**Current workaround:**
g_eval.py 
Not setting logprobs and top_logprobs   self.model.a_generate_raw_response and  self.model.generate_raw_response (top_logprobs is configurable already but not logprobs)

gpt_model.py
Setting the temperature to 1
```
        else:
           # check necessary, if model_name = o3-mini set temperature=1
            return ChatOpenAI(
                model_name=self.model_name,
                openai_api_key=self._openai_api_key,
                base_url=self.base_url,
                *self.args,
                **self.kwargs
            )

```

With these changes o3-mini (and also o1-mini if manually added to the list of supported models) works with deepeval.

Facing the same issue. o3 mini seems to be more suitable due to its reasoning capabilities for test evaluation. Thanks @RicardoBochnia for the work around.

I see that issue with o4-mini.
I use it as simple as:

`GPTModel(model="o4-mini")`

and also:

`GPTModel(model="o4-mini", logprobs=False, top_logprobs=0, temperature=1.0)
`

And get: `openai.PermissionDeniedError: Error code: 403 - {'error': {'message': 'You are not allowed to request logprobs from this model', 'type': 'invalid_request_error', 'param': None, 'code': None}}`

Any idea ?

BTW, o3-mini worked for me....
Repository: confident-ai/deepeval
Issue: Regression in faithfulness metric (invalid key)
Root cause: Validation Gap

It seems that a regression appears recently in the faithfulness metric. It concerns the prompt template where the key "claims" is using in the example instead of "truths". This brakes the computation as the llm output is never compliant with schema.

Here is the source file involved:
https://github.com/confident-ai/deepeval/blob/22692bf630a399a83596d8d489e3150fc2fe4a02/deepeval/metrics/faithfulness/template.py#L36

Thanks its solved now @LudoBar
Repository: confident-ai/deepeval
Issue: Verbose printing goes into infinite loop in notebook
Root cause: Resource Mishandling

Looking for some help, a long running tricky bug:

In colab notebooks, when you turn `verbose_mode` = `True` what happens is you get a recursion error. This isn't reproducible when running the methods to print verbose logs in isolation.

See the code here: https://colab.research.google.com/drive/10zfwulUlTOXpJibVAjUGy-cdybi2JsLx#scrollTo=PvIxWTuU1aI4

Looking into this. Will keep you updated.

@penguine-ip this needs to be closed Ig

@jayzalani yup!
Repository: confident-ai/deepeval
Issue: Request timeout when using multiple metrics
Root cause: Documentation Deficiency

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**

I am building an evaluation with multiple metrics

`def evaluate_llm_response(test_case):
    return evaluate(
        test_case,
        [hallucination_metric, relevancy_metric, bias_metric]
    )`

I am using gpt4o for evaluation, and always receive the error message `APITimeoutError: Request timed out.`. If I reduce the number of metrics to 2, it works well.

**To Reproduce**
Run the function `evaluate_llm_response` with any test case

**Expected behavior**
All metrics are evaluated

**Screenshots**


**Desktop (please complete the following information):**
 - OS:MacOS


**Additional context**
Add any other context about the problem here.


@vinhqdang maybe your evaluation llm ran into rate limits?

have the same issue, I need to batch the dataset (into batches of 5) to be able to run and even then it runs with a few retries

I also encountered this issue. To address it, I split the total evaluation where my chunk size were 20. Although this approach takes a bit more time to complete, it allows the evaluation to run without getting stuck.

Hi, the same issue. Can anything be done with it?

I'm also experiencing this issue. Has anyone had any success with the [DeepEval Retry / Backoff Tuning options](https://deepeval.com/docs/evaluation-flags-and-configs#retry--backoff-tuning)?

Hey @vinhqdang @chiral-carbon @AfrarJahin @vaclcer @JoeMountWSWS there's actually an option of setting the `max_concurrent` and `throttle` values to not run evals all at once: https://deepeval.com/docs/evaluation-flags-and-configs#async-configs

^The docs above some how to use `async_configs`
Repository: confident-ai/deepeval
Issue: Unable to run Ragas metrics, evaluate ends with TypeError: capture_metric_type() missing 1 required positional argument: 'async_mode'
Root cause: Interface Contract Mismatch

Dears,
Using RAGAS metrics for evaluation,  error appears:  TypeError: capture_metric_type() missing 1 required positional argument: 'async_mode'

Steps to reproduce:
1. Get example from: https://docs.confident-ai.com/docs/metrics-ragas#example
2.  Install Deepeval, Ragas, set OPENAI_API_KEY

```
pip install ragas
...
deepeval==2.4.1
ragas==0.2.13
```

3.  Run example:
```
from deepeval import evaluate
from deepeval.metrics.ragas import RagasMetric
from deepeval.test_case import LLMTestCase

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

# Replace this with the expected output from your RAG generator
expected_output = "You are eligible for a 30 day full refund at no extra cost."

# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = ["All customers are eligible for a 30 day full refund at no extra cost."]

metric = RagasMetric(threshold=0.5, model="gpt-3.5-turbo")
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output,
    expected_output=expected_output,
    retrieval_context=retrieval_context
)

metric.measure(test_case)
print(metric.score)

# or evaluate test cases in bulk
evaluate([test_case], [metric])
```

Whole Output:

```
_.py:53: UserWarning: You are using deepeval version 2.4.1, however version 2.4.2 is available. You should consider upgrading via the "pip install --upgrade deepeval" command.
  warnings.warn(
Traceback (most recent call last):
  File "/Volumes/Store/perpetual/deepeval_trys/ragas_aync_error.py", line 22, in <module>
    metric.measure(test_case)
  File "/Volumes/Store/perpetual/deepeval_trys/venv/lib/python3.11/site-packages/deepeval/metrics/ragas.py", line 503, in measure
    with capture_metric_type(self.__name__):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 289, in helper
    return _GeneratorContextManager(func, args, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 105, in __init__
    self.gen = func(*args, **kwds)
               ^^^^^^^^^^^^^^^^^^^
TypeError: capture_metric_type() missing 1 required positional argument: 'async_mode'
```


Could you give an instruction how to deal with this?



Adding async_mode=False as a parameter to every call of capture_metric_type in deepeval/metrics/ragas.py should fix it, I think?

believe it has been solved, closing this now
Repository: confident-ai/deepeval
Issue: non-conversational metrics, when applied to a ConversationalTestCase, will NOT evaluate the last turn
Root cause: Algorithmic Error

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
Documentation states: non-conversational metrics (which are metrics used for individual LLMTestCases), when applied to a ConversationalTestCase, will evaluate the last turn in a 
While in `AnswerRelevancyMetric` and other 13 metrics, not the last, but first one is evaluated:
```
if isinstance(test_case, ConversationalTestCase):
     test_case = test_case.turns[0]
```


**To Reproduce**
not required

**Expected behavior**
sync documentation and implementation

**Screenshots**

**Desktop (please complete the following information):**

**Smartphone (please complete the following information):**

**Additional context**
Add any other context about the problem here.


@sermakarevich good catch....can't believe this got unnoticed for so long

@sermakarevich its fixed!
Repository: confident-ai/deepeval
Issue: red teaming metrics are broken for conversation case
Root cause: Validation Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**

there are two issues in one place: 
```
        if isinstance(test_case, ConversationalTestCase):
            test_case = check_conversational_test_case_params(test_case, self)
        check_llm_test_case_params(test_case, required_params, self)
```
while 
```
def check_conversational_test_case_params(
    test_case: ConversationalTestCase,
    test_case_params: List[LLMTestCaseParams],
    metric: BaseConversationalMetric,
    require_chatbot_role: bool = False,
):
```
1. `check_conversational_test_case_params` has 3 required params but passed 2 
2. `check_conversational_test_case_params` does not return anything, so test_case is None-ified if it is a ConversationalTestCase


@sermakarevich the red teaming metrics aren't meant to be used externally but if you make a PR happy to potentially incorperate it
Repository: confident-ai/deepeval
Issue: `include_reason` is not taken care of while using the metrics.
Root cause: Validation Gap

**Describe the bug**
A clear and concise description of what the bug is.

Deepeval version: 2.4.7

**To Reproduce**
Steps to reproduce the behavior:
```py
metric = ConversationRelevancyMetric(model=model, threshold=0.5, include_reason=False)

metric.reason is not None or ''
```

**Expected behavior**
If `include_reason=False`, it should not compute the reason. But here you can see it is still computing the reason and not taking `include_reason` into consideration.

https://github.com/confident-ai/deepeval/blob/17c3e4f78c1c0d7eef4652051796a34875d5c94a/deepeval/metrics/conversation_relevancy/conversation_relevancy.py#L81-L83
https://github.com/confident-ai/deepeval/blob/17c3e4f78c1c0d7eef4652051796a34875d5c94a/deepeval/metrics/conversation_relevancy/conversation_relevancy.py#L160-L182




@rohitgr7 it now is! thanks
Repository: confident-ai/deepeval
Issue: Confusing tools in the Tool Correctness metric reason
Root cause: Algorithmic Error

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
When calling the `measure` method on the `ToolCorrectness` metric using the `should_exact_match=True` parameter in `metric.reason`, a message with mixed up tools is displayed.

**To Reproduce**
```
tool_correctness = ToolCorrectnessMetric(
    should_exact_match=True, verbose_mode=True, evaluation_params=[ToolCallParams.TOOL]
)

test_case = LLMTestCase(
        input="What if these shoes don't fit?",
        actual_output="We offer a 30-day full refund at no extra cost.",
        # Replace this with the tools that was actually used by your LLM agent
        tools_called=[ToolCall(name="WebSearch", input_parameters={}), ToolCall(name="ToolQuery", input_parameters={})],
        expected_tools=[ToolCall(name="WebSearch", input_parameters={})],
    )
    
tool_correctness.measure(test_case)
print(tool_correctness.score)
print(tool_correctness.reason)
```

**Expected behavior**
In the output `Reason`, the expected tools should go first, then the called tools.

**Screenshots**

![Image](https://github.com/user-attachments/assets/efe2436f-e120-46f2-a083-70d31b7b82fd)

**Additional context**
deepeval version - 2.3.3
Repository: confident-ai/deepeval
Issue: Write to local directory when write_cache = False
Root cause: Validation Gap

Bug:
When running in an Azure Function App or AWS Lambda: 
evaluation_results = evaluate(test_cases=test_cases, metrics=metrics, ignore_errors=True, write_cache=False)

write_cache =False the evaluate function still attempts to write to local file. 

[Errno 30] Read-only file system: 'temp_test_run_data.json' -> '20250304_113435'

Any solution on this ? 
Thanks. 

You additionally need to disable telemetry, so it won't write anything. See here https://github.com/confident-ai/deepeval/issues/975#issuecomment-3052576693