zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: GEval Score above 1.0?
Root cause: Algorithmic Error



geval_metric = GEval(
        name="Recipe translation quality",
        criteria="Determine whether the actual output is factually correct based on the expected output",
        evaluation_steps=[
        "Compare the 'actual output' directly with the 'expected output' to determine if the main answer aligns factually without focusing on length.",
        "Consider only the main answer's factual content in 'actual output' and ignore any additional details, reasoning, or verbosity beyond the expected output.",
        "Ensure the 'actual output' does not introduce any factual errors or contradictions in relation to the 'expected output'.",
        "Treat 'expected output' as the ideal and only correct answer; do not reference any external knowledge or other answers in scoring.",
        "Do not penalize for missing explanation as long as the main factual answer in 'actual output' is accurate and agrees entirely with the 'expected output'."
        ],
        evaluation_params=[
            LLMTestCaseParams.INPUT,
            LLMTestCaseParams.ACTUAL_OUTPUT,
            LLMTestCaseParams.EXPECTED_OUTPUT,
        ],
        model=evaluator_llm
    ) 



geval_score=geval_metric.measure(test_case)
geval_score_reason=geval_metric.reason
print(f"GEval Score: {geval_score:.4f} | Reason: {geval_score_reason}")

**Output:
⠹ ✨ You're running DeepEval's latest Recipe translation quality (GEval) Metric! (using deepseek-r1:32b, strict=False, async_mode=True)...HTTP Request: POST http://localhost:11434/api/generate "HTTP/1.1 200 OK"
GEval Score: 10.0000 | Reason: The translation adheres to the evaluation steps provided, with only minor variations in phrasing that do not affect accuracy.**






Suprisingly if i set strict_mode=True i do not get this issue ⠏ ✨ You're running DeepEval's latest Recipe translation quality (GEval) Metric! (using deepseek-r1:32b, strict=True, async_mode=True)...HTTP Request: POST http://localhost:11434/api/generate "HTTP/1.1 200 OK"
GEval Score: 1.0000 | Reason: All evaluation steps were followed correctly.
Repository: confident-ai/deepeval
Issue: Allow score in MetricData to accept string values
Root cause: Unimplemented Feature Gap

Currently, in the MetricData class, the score field is strictly typed as an Optional[float]:

```python
class MetricData(BaseModel):
    ...
    score: Optional[float] = None
    ...

```

However, we have encountered use cases where the LLM returns a score as a string, such as:

"acceptable"

"ok"

"not acceptable"

"True"

These are semantically meaningful and useful in contexts where the model output isn't strictly numerical.

Why this is worth fixing:
The current strict typing leads to a TypeError in the evaluate() method when using custom metrics that return strings. This creates unnecessary friction when integrating non-numeric model outputs into evaluation pipelines.

We would like to propose updating the type of score to:

```python
score: Optional[Union[float, str]] = None
```

This change would maintain backward compatibility while allowing for more flexible and descriptive scoring outputs from LLMs.

Happy to submit a PR!


Hey @raphdoo we'll have to think about this first since it would involve a lot of updates to the BaseMetric class (we don't want score to be both float and string)

I'm wondering are you using this for G-Eval or what metric?

I've been working on a type cleanup PR for deepeval that a change like this would really complicate. It muddies the definition of what score means and would cause a lot of headache throughout the app for checking the type of score before using it (e.g., how does this work in strict mode? what does threshold do now?)

> I'm wondering are you using this for G-Eval or what metric?

I am using it for a multiclass string-based custom metric which we have found as the best use case compared to numerical based scoring. An example custom prompt defined;

""" Using the following guideline, score the actual_output. Only return the score choosing one from Unacceptable, Good and Excellent""". 

The guideline then provide structure definitions for Uncceptable, Good and Excellent which forms part of the standard input to the prompt.

> I've been working on a type cleanup PR for deepeval that a change like this would really complicate. It muddies the definition of what score means and would cause a lot of headache throughout the app for checking the type of score before using it (e.g., how does this work in strict mode? what does threshold do now?)

Yeah, I get this and I think it might complicate things a little, but could also provide flexibility on scoring. 

For custom metric where multi-class string-based scoring is used, strict mode is set to false which I think should be the expected behaviour.

The threshold will have to be updated to not only support numbers but also an array of string. For instance, if my multiclass prompt expects one of the three outputs "Unacceptable", "Bad", "Good", and "Excellent". We can define our acceptance criteria threshold as only scores that are either "Good" or "Excellent". A simple way to refactor this could be threshold supporting array data type such that threshold can be set as an array of Acceptable strings i.e ["Good", "Excellent"].
Repository: confident-ai/deepeval
Issue: Gemini model classes should not hardcode valid models
Root cause: Architectural Constraint

Currently, the `gemini_model.py` in `llms` and `mlllms` folders hardcode a list of valid model names and use that list for error checking. This is good for error checking but it's very brittle as new models come up almost every month and old models get deprecated. The list is already out of date as of today.

My suggestion is to not hard code this list at all. If a user enters the wrong or out-of-date model, the request will fail anyway. It might be harder to debug for the user why the request failed but keeping this list up to date and relying on a new release of DeepEval for every model update is not going to work either.
Repository: confident-ai/deepeval
Issue: Issue instantiating TopicalGuard
Root cause: Documentation Deficiency

**Describe the bug**
Cannot instantiate TopicalGuard as described in the guide and the tests

**To Reproduce**
>>> import deepeval
>>> deepeval.__version__
'2.7.2'
>>> topical_guard = TopicalGuard(
...     allowed_topics=["healthcare", "education", "finance"]
... )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: TopicalGuard.__init__() got an unexpected keyword argument 'allowed_topics'

**Additional context**
This is my first time trying out deepeval - excited about where this project is going

Hi Patrick, are you still running into this issue?

@penguine-ip , I am looking through the documentation -  I do not believe this is the correct way to use the guard correct?

The constructor for TopicalGuard doesn't seem to have a parameter for allowed topics
Repository: confident-ai/deepeval
Issue: Traces Observatory Disappeared
Root cause: Documentation Deficiency

**Describe the bug**
In the DeepEval Platform, in the observatory section, the traces disappeared.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to https://app.confident-ai.com/
2. Click on 'Observatory'
3. Go to 'Traces'
4. "No traces found"

**Expected behavior**
The expected behavior is to see the traces.

**Screenshots**
![Image](https://github.com/user-attachments/assets/e0cae948-3b66-4bea-8934-49ef7e30955a)

**Desktop (please complete the following information):**
 - OS: Windows
 - Browser: Brave
 - Version: 1.77.97

Hey @SnowArtz it got moved to a different implementation: https://documentation.confident-ai.com/llm-observability/llm-tracing
Repository: confident-ai/deepeval
Issue: Error reported when using tool correctness metric
Root cause: Documentation Deficiency

**Describe the bug**
Error reported when using tool correctness metric

Error：
`evaluation_params=[ToolCallParams.TOOL, ToolCallParams.INPUT_PARAMETERS],
                       ^^^^^^^^^^^^^^^^^^^
AttributeError: type object 'ToolCallParams' has no attribute 'TOOL'`

Code：
`from deepeval.metrics import ToolCorrectnessMetric
from deepeval.test_case import LLMTestCase, ToolCallParams, ToolCall

test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output="We offer a 30-day full refund at no extra cost.",
    tools_called=[ToolCall(name="WebSearchTool"), ToolCall(name="QueryTool")],
    expected_tools=[ToolCall(name="WebSearchTool")]
)

metric = ToolCorrectnessMetric(
  evaluation_params=[ToolCallParams.TOOL, ToolCallParams.INPUT_PARAMETERS],
  should_consider_ordering=True
)

metric.measure(test_case)
print(metric.score)
print(metric.reason)`




![Image](https://github.com/user-attachments/assets/1b82076a-ceec-4ab8-a368-1525f39519d2)

Hey @hsx1612727380 
Yes ToolCallParams doesn't have any attribute 'TOOL'
Did you write this code snippet on your own or did you get it from somewhere?

Here are some docs on how to use tool correctness metrics for your reference [https://www.deepeval.com/docs/metrics-tool-correctness](url)

> Hey [@hsx1612727380](https://github.com/hsx1612727380) Yes ToolCallParams doesn't have any attribute 'TOOL' Did you write this code snippet on your own or did you get it from somewhere?
> 
> Here are some docs on how to use tool correctness metrics for your reference [https://www.deepeval.com/docs/metrics-tool-correctness](url)

The link I referenced is https://www.confident-ai.com/blog/llm-agent-evaluation-complete-guide

![Image](https://github.com/user-attachments/assets/7ba26eeb-9e09-470a-a35c-e223ee4e2f8c)
Repository: confident-ai/deepeval
Issue: Calculating metric scores with evaluate blocks even when using async_mode = False and disabling caches
Root cause: Resource Mishandling

Hello; as per the subject when using evaluate on 300 test_cases and evaluating say, 4 metrics with modell llama3.2:latest from Ollama, the process blocks at the very end of the computation.

I have disabled writing files of caching in evaluate().

I am using a GPU on Windows 11, using as I said Ollama llama3.2:latest.

Can you please fix this behaviour? 

The same behaviour is way more evident with async_mode = True.

@GiambaDS86 it might be your model getting stuck actually, do you have test cases with really long piece of text?
Repository: confident-ai/deepeval
Issue: Asynchronous/Synchronous behavior and documentation of ConversationSimulator not correct
Root cause: Interface Contract Mismatch

**Describe the bug**
ConversationSimulator does not work when `async_mode=False`.
ConversationSimulator expects async CoRoutine instead of (str) -> str callback when `async_mode=True`


**To Reproduce**
Copy paste the example from the ConversationSimulator documentation. Alter `async_mode=False`. Note the following error:
```    async def _a_generate_chatbot_response(
        self, input: str, model_callback: Callable
    ):
>       res = await model_callback(input)
E       TypeError: object str can't be used in 'await' expression```

Comment out or set `async_mode=True`. Run again. Note a separate but related error:
```    def _simulate_first_user_input(
        self, scenario: str, user_profile: str
    ) -> str:
        prompt = ConversationSimulatorTemplate.generate_first_input(
            scenario, user_profile, self.language
        )
    
        if self.using_native_model:
>           res, cost = self.simulator_model.a_generate(
                prompt, schema=SimulatedInput
            )
E           TypeError: cannot unpack non-iterable coroutine object```

(branching may cause slight differences depending on setup)
In both cases, the expected signature of the function call will not be correct.

**Expected behavior**
If model_callback is a `(str) -> str` callback, it cannot be awaited.
If `async_mode=False`, model callback should not be a co-routine and the `a_*` internal methods should be replaced by `*` methods with synchronous implementations (i.e. generate instead of `a_generate`).
The example use case of ConversationSimulator from the documentation should be updated to be defined as a coroutine/asynchronous function, the async_mode=False behavior should be working, and the difference documented.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: MacOS
 - Browser N/A
 - Version N/A

**Additional context**
Using deepeval 2.7.5


@marr75 fixed and upcoming in the coming release!

Awesome! This feature is so useful even with the 2.7.5 quirks.

Hey @marr75 is fixed, can you see if it works?
Repository: confident-ai/deepeval
Issue: AttributeError: 'tuple' object has no attribute 'answer'
Root cause: Interface Contract Mismatch

**Describe the bug**
When I tried to evaluate the Ollama model using the MMLU (Massive Multitask Language Understanding) from deepeval, the following error occurred:**AttributeError: 'tuple' object has no attribute 'answer'**

`from deepeval.models import OllamaModel
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.benchmarks import MMLU
from deepeval.benchmarks.mmlu.task import MMLUTask

model = OllamaModel(
    model="qwen2.5:7b",
    base_url="http://localhost:11434"
)
answer_relevancy = AnswerRelevancyMetric(model=model)
print(model.generate("你好"))

benchmark = MMLU(
    tasks=[MMLUTask.HIGH_SCHOOL_COMPUTER_SCIENCE, MMLUTask.ASTRONOMY],
    n_shots=3
)

benchmark.evaluate(model=model)
print(benchmark.overall_score)`
**Screenshots**

![Image](https://github.com/user-attachments/assets/5f286536-e1ce-4c8e-b886-42f5e16d2d6e)

![Image](https://github.com/user-attachments/assets/27f98c70-60d3-44ef-bb37-1c4dd7869907)

**Desktop (please complete the following information):**
 - OS: ubuntu 24.04
 - deepeval 2.7.6, install by pip

Hi @kunlun122 , Please find the fix in the PR. you can clone the repo and edit the code. This will help you to run the code.
Repository: confident-ai/deepeval
Issue: Add SQL Evaluation
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Add a way to evaluate SQL queries based maximizing info gain while minimizing number of rows for synthetic query generation. Minimizing number of rows is important for 

**Describe the solution you'd like**
Warning- this API is a WIP. Very open to suggestions. 

```python

from deepeval.sql import SQLEval
table = SQLEval.load_table(...)
```

**Describe alternatives you've considered**
Can't really see other alternatives for SQL tables right now. 

**Additional context**
May require a bit of work around building SQL injection and also providing a frontend to make viewing the created table very simple.
Repository: confident-ai/deepeval
Issue: Deepeval 2.7.* incompatible with poetry 2.*
Root cause: External Dependency Breakage

**Describe the bug**
Can not install deepeval>=2.7 alongside poetry>=2 due to version conflict for `pkginfo` dependency.

Dependency chains:
- poetry <- pkginfo<2.0,>=1.12
- deepeval <- twine==5.1.1 <- pkginfo<1.11

**To Reproduce**
Steps to reproduce the behavior:
1. Create a new Python environment
2. Run `pip install "poetry>=2" "deepeval>=2.7"`
3. Command will fail with error: `ERROR: Cannot install deepeval and poetry==2.0.0 because these package versions have conflicting dependencies.`

**Expected behavior**
The install should run without package version conflicts.

**Desktop:**
 - OS: Windows 11
 - Deepeval: >=2.7.*
 - Python: 3.11.9

**Additional context**
Though the actual conflict is caused by the `twine` package, I believe this is an issue that should be addressed by deepeval, because the current version constraint for the dependency on twine is very narrow.

If possible, the issue should be addressed by relaxing the dependency on twine or by following best practice and splitting the deepeval dependencies into strictly required dependencies and dev dependencies, such that end-users of deepeval only have to install the dependencies required to run the package and not install unnecessary dependencies only required for development.


@robin-mader-bis Fixed in latest 2.8.3, moved twine to dev, can you try again and see if it works?

@penguine-ip Just got around to testing it, works as expected now 👍

Thank you for the fast fix!
Repository: confident-ai/deepeval
Issue: The input parameter for G-Eval
Root cause: Documentation Deficiency

**Background:** 
My study involves generating textual summaries for function-level binary code. However, given the poor performance of traditional token-based metrics(BLEU, METEOR, ROUGE, etc.), evaluating such summaries remains a challenge.

**My task:** 
Use a large language model (LLM) to evaluate code summaries generated by existing work.

**The expected workflow:** 
Using G-Eval, provide the function’s source code, original summary, and decompiled code, along with evaluation criteria. Then, supply the output generated by a particular existing work for that function, and finally obtain an evaluation result (e.g., a score).
Currently, I tend to treat the reference facts (i.e., source code, original summary, and decompiled code) as the **context** parameter, and the output from a given work as the **ACTUAL_OUTPUT**.

**My question:** 
Since it has been pointed out that "input" is a necessary parameter, I’m confused about how to specify the input. In the BinaryCodeSummary task, most current methods rely on embedding models(**not LLM application**), and the model inputs often cannot be represented in text form—let alone clearly specified.

Is G-Eval suitable for my task? Do you have any suggestions? I sincerely look forward to your guidance.

By the way, what parameter should I declare to print a view of the prompt that is eventually sent to LLM?

hey @linjikun1 can you please ask in our discord, its better for conversation there, thanks!
Repository: confident-ai/deepeval
Issue: Typo in azure_model.py: `deploynment_name` should be `deployment_name`
Root cause: Validation Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Instantiate the model with the typo’d parameter:
  ```python
  model = AzureOpenAIModel(
    model_name="gpt-4o",
    azure_openai_api_key=...,
    openai_api_version="2024-09-01-preview",
    azure_endpoint="https://your-endpoint/",
    temperature=0,
    deploynment_name="gpt-4o",   # ← typo here
)
  ```
2. Use that model when measuring answer relevancy:
  ```python
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.tests import LLMTestCase

metric = AnswerRelevancyMetric(
    threshold=0.5,
    model=model,
    include_reason=True
)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output="We offer a 30-day full refund at no extra cost."
)
metric.measure(test_case)
print(metric.score, metric.reason)
  ```
3. Observe the error thrown during `metric.measure(...)`
   **Error**
  ```
  AttributeError: 'AzureOpenAIModel' object has no attribute 'deploynment_name'
  ```

**Expected behavior**
The `AzureOpenAIModel` should correctly accept and store the `deployment_name` parameter, so that downstream metrics like `AnswerRelevancyMetric` run without raising an `AttributeError`.

**Additional context**
As a temporary workaround, you can copy the correct value onto the misspelled attribute after initialization:
```python
model.deploynment_name = model.deployment_name
```
I haven’t tried it yet, but it will probably fix the issue if you correct the typo at line 282 of the source:
```diff
-    deploynment_name=self.deployment_name,
+    deployment_name=self.deployment_name,
```
https://github.com/confident-ai/deepeval/blob/1b0563b01601e2d89bc34dcbe9d7bdfbb9a018e0/deepeval/models/llms/azure_model.py#L282

@naribow released!
Repository: confident-ai/deepeval
Issue: Versioning
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
If you are an engineer, it would be really important to be able to version and compare results as if it were a git branch

**Describe the solution you'd like**

```
git checkout -b feature/add-guidance

# To compare this against the most recent branch in terms of performance (which is saved/cached)
deepeval compare

# To compare the main branch
deepeval compare main 
```

Ideally it should then output a table to compare results
Repository: confident-ai/deepeval
Issue: TypeError: evaluate() takes 0 positional arguments but 2 were given

**Describe the bug**
TypeError: evaluate() takes 0 positional arguments but 2 were given

**To Reproduce**
results = evaluate(test_data, metrics)

https://www.deepeval.com/docs/evaluation-running-llm-evals#configs-for-evaluate
this page is not available

Update: My function name was also evaluate, so thats the reason I got this error
Repository: confident-ai/deepeval
Issue: Error: [Errno 30] Read-only file system: '.deepeval'
Root cause: Validation Gap

facing this error while trying to use the library in aws lambda.

@sambit417 try opting out for telemetry, it is trying to write to file: https://www.deepeval.com/docs/data-privacy

After upgrading from 2.7.6 to 3.0.3, I'm running into the same issue. In my lambda, I get the error:

```
  File "/var/lang/lib/python3.12/site-packages/deepeval/telemetry.py", line 39, in <module>
    os.makedirs(HIDDEN_DIR, exist_ok=True)
  File "<frozen os>", line 225, in makedirs
```

Looks like [telemetry.py](https://github.com/confident-ai/deepeval/blob/main/deepeval/telemetry.py):33-50 tries to do some file system manipulation when it is imported. I don't see any check for `DEEPEVAL_TELEMETRY_OPT_OUT` that would prevent this code from running. These changes were introduced [here](https://github.com/confident-ai/deepeval/blob/8f5a287f3c6a4dd3ffb77e07afb89af98e0ac852/deepeval/telemetry.py).

+1 for supporting read-only file systems. I can try to roll a fix for this if it would be helpful. Maybe this code should only be executed, if `DEEPEVAL_TELEMETRY_OPT_OUT` is not set to "YES".

Tested #1654. Seems to fix the issue on my end.
Repository: confident-ai/deepeval
Issue: AttributeError: 'MMLU' object has no attribute 'shots_datase'. Did you mean: 'shots_dataset'?
Root cause: Validation Gap

**Describe the bug**

typo in MMLU. Prevents completion of the benchmark.
it's at this line: https://github.com/confident-ai/deepeval/blob/a6254b1e2fd33cf6e5c0f25d32f84b7ca119a419/deepeval/benchmarks/mmlu/mmlu.py#L160C18-L160C30

**To Reproduce**
Steps to reproduce the behavior:
1. Create a simple MMLU benchmark
2. Run it and it will panic

**Expected behavior**
Benchmark completetion

@Posrabi its released!
Repository: confident-ai/deepeval
Issue: QA with your doc

I think the documentation for DeepEval is already well-organized, but it would be even more helpful if it supported AI-powered question-answering capabilities within the doc, similar to what Milvus or VLLM offers.

Any good recommendations?

@penguine-ip  https://inkeep.com/ : this is what milvus, anthropic, pinecone and other leading teams are using.

https://www.runllm.com/ : this is used by databricks, llamaindex, dspy and some other good orgs.
although, u always have the option to build it in-house also :p 

@abhishekrp-ai2002 one thing we 100% need is visibility into what users are asking, do u know if these allow so?

@penguine-ip  i think both of them support insights and analytics which traces each input query ( https://docs.runllm.com/dashboard and https://docs.inkeep.com/analytics/usage-reports )
another workaround or fallback could be to add a gateway / proxy like service from your end which traces all the input-output pairs ( maybe something like [LiteLLM](https://www.litellm.ai/) )