zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: The latest versions of deepeval and Instructor can't be installed together
Root cause: External Dependency Breakage

**To Reproduce**
Steps to reproduce the behavior:
1. `pip install deepeval==1.4.1 instructor==1.6.2`
2. See error

**Expected behavior**
Both libraries install

**Screenshots**
```
Collecting instructor==1.6.2
  Downloading instructor-1.6.2-py3-none-any.whl.metadata (17 kB)
Collecting deepeval==1.4.1
  Downloading deepeval-1.4.1-py3-none-any.whl.metadata (977 bytes)
ERROR: Cannot install deepeval==1.4.1 and instructor==1.6.2 because these package versions have conflicting dependencies.

The conflict is caused by:
    deepeval 1.4.1 depends on tenacity~=8.4.1
    instructor 1.6.2 depends on tenacity<10.0.0 and >=9.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
```

The [latest version](https://github.com/jd/tenacity/releases/tag/9.0.0) of Tenacity made a breaking change to the statistics attribute. I did a quick search in deepeval and couldn't find an obvious usage of that attribute.

Same. It could be much helpful to have a fix on that.

same

@stevermeister  @yuriykuzin  @milest I tried tenacity ^9.0.0 but it will be conflicting with langchain.

From poetry:
```
... because langchain (0.1.6) depends on tenacity (>=8.1.0,<9.0.0)
```

I think the better solution would be to downgrade instructor?

Ah, I see. [langchain==0.1.6](https://github.com/langchain-ai/langchain/releases/tag/v0.1.6) is a fairly old version, isn't it? The most recent release is [0.3.6](https://github.com/langchain-ai/langchain/releases/tag/langchain%3D%3D0.3.6).

Downgrading Instructor is definitely an option. Would another option be to require a newer version of langchain?

Looks like even `langchain 0.3.7` still requires older tenacity 😞 
```
And because deepeval depends on langchain (^0.3.7), 
every version of llama-index-llms-langchain requires tenacity (>=8.2.0,<9.0.0).
```
Repository: confident-ai/deepeval
Issue: HumanEval doesn't actually runs test
Root cause: Unimplemented Feature Gap

HumanEval.evaluate checks only  if generated sample is correct python. 
It doesn't actually run tests like the original project does.
https://github.com/confident-ai/deepeval/blob/f8d5f8fe51ddf6a07b6eadc2117002a9eec65f26/deepeval/benchmarks/human_eval/human_eval.py#L96-L97


Up, the problem is still here

An example of generated 'function' here:

```
from typing import List

def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """
    
    for i in range(len(numbers) - 1):
        current_distance = numbers[i] - numbers[i + 1]
        if current_distance < threshold:
            return True
    
    return False
```

And then example of `golden.expected_output`

```

METADATA = {
    'author': 'jt',
    'dataset': 'test'
}

def check(candidate):
    assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.3) == True
    assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.05) == False
    assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.95) == True
    assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.8) == False
    assert candidate([1.0, 2.0, 3.0, 4.0, 5.0, 2.0], 0.1) == True
    assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 1.0) == True
    assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 0.5) == False
```

This code obviously won't do anything. Moreover, current implementation of:

https://github.com/confident-ai/deepeval/blob/f8d5f8fe51ddf6a07b6eadc2117002a9eec65f26/deepeval/benchmarks/human_eval/human_eval.py#L93-L102

1. Is unsafe, and can harm your computer if LLM got crazy
2. Doesn't handle Python exceptions if LLM generated malformed Python code

Stumbled upon the same issue, HumanEval isn't, in fact, supported

+1
Repository: confident-ai/deepeval
Issue: Enable the discussion feature on the repository

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Is your feature request related to a problem? Please describe.**

My problem is that I do not have a discord account and do not want to create one but have a question.

**Describe the solution you'd like**

Open the discussion section on the repo. 
Discussion gives a clear way of working people are used to in term of features (search, notifications etc).
Discussions do not require an account on another platform.
If you prefer the discord group, you can still use it in addition. With the discussions, other users can jump in and answer questions on github.


**Additional context**

I am on a professional setup.

https://github.com/confident-ai/deepeval/discussions/1115
Repository: confident-ai/deepeval
Issue: Get tokens for each request
Root cause: Unimplemented Feature Gap

hello
In deepeval, I can get tokens use metric.evaluation_cost, but the process excludes some models and the prices are openai prices. If I use other products' models, how do I get tokens and then calculate the cost myself?
However, I didn't find any exposed fields in metric

@luqianjiang there's no way to do it right now, if you can open a PR for it we can support it
Repository: confident-ai/deepeval
Issue: Add `tools_called_key_name` and `expected_tools_key_name` to both .add_test_cases_from_json_file() and .add_test_cases_from_csv_file() methods
Root cause: Unimplemented Feature Gap

I find it to be such an easy and straightforward fix, and it is frustrating to have to change the library code in order to make it work for my case and I imagine many other people would appreciate this addition.

I feel like it would make sense for the .add_test_cases_* methods to include all parameters of the constructor for the LLMTestCase class, hence why I am writing this issue (which I hope gets resolved soon) in order to just add these few parameters in said methods.

Attached below is a screenshot of what I attempted to do as a quick fix, however you are more aware of the effect this has on the codebase. If it breaks something somewhere else, a similar solution would probably take not much more effort than changing a few lines.

![image](https://github.com/user-attachments/assets/d392948b-c522-42c5-ac1f-b89db35e7055)



@rayenf-mw its out!
Repository: confident-ai/deepeval
Issue: utf-8 not supported [with solution]
Root cause: Validation Gap


**Describe the bug**
Utf-8 characters are not properly saved to files when using synthesizer.save(), here is a simple solution:

solution in synthesizers.py

```
        if file_type == "json":
            with open(full_file_path, "w", encoding="utf-8") as file:
                json_data = [
                    {
                        "input": golden.input,
                        "actual_output": golden.actual_output,
                        "expected_output": golden.expected_output,
                        "context": golden.context,
                        "source_file": golden.source_file,
                    }
                    for golden in self.synthetic_goldens
                ]
                json.dump(json_data, file, indent=4, ensure_ascii=False)
        elif file_type == "csv":
            with open(full_file_path, "w", newline="", encoding="utf-8") as file:
```
Repository: confident-ai/deepeval
Issue: 'Answer Length' metric
Root cause: Unimplemented Feature Gap

aka 'Brevity' metric.  Simple character count of answer (this metric might be somewhere else in the code but I'm not seeing it?)  Some models could score higher on relevancy but with more words (in theory) - an 'answer length' metric would control for that.

This is a great point. I like this metric - i added it as a custom metric with a very simple implementation here but never thought about having it as an official metric but I can see the use case for it. Feel free to submit a PR - would love to take a look! 
https://docs.confident-ai.com/docs/quickstart/custom-metrics


PR here - looks like it failed a lint formatting curious what you're using and what I could run it by locally - wasn't sure how to incorporate other than just what it is - used same as your custom metric but checks for > 1 character and returns 'AnswerLength' 
https://github.com/confident-ai/deepeval/pull/118

This was merged and closed :)
Repository: confident-ai/deepeval
Issue: faithfulness print extraction_limit for each test case
Root cause: Validation Gap

When run evaluate with faithfulness metrics, the template print extraction_limit for each test case. which is very annoying. The print it due to line 36 at deepeval/metrics/faithfulness/template.py:

print(extraction_limit)

I don't see why this is needed, should be remove.

@rjiangnju its just a mistake
Repository: confident-ai/deepeval
Issue: OpenAI Rate Limit Exceeds
Root cause: Resource Mishandling

I am trying to run evaluate on my dataset with one metric (relevancy) and imeeidtaely get errors on getting to OpenAI rate limits, how to solve it? 

Event loop is already running. Applying nest_asyncio patch to allow async execution...
Evaluating 80 test case(s) in parallel: |          |  0% (0/80) [Time Taken: 00:00, ?test case/s]ERROR:root:OpenAI rate limit exceeded. Retrying: 1 time(s)...

I think you have to see whether your openai is being used elsewhere because one metric run wouldn't cause a rate limit error

I got the same issue as@miriam2040. Initially it was running fine, then suddenly i get this error.
Repository: confident-ai/deepeval
Issue: Hallucination metric assigns only either 0.0 or 1.0 score
Root cause: Algorithmic Error

**Describe the bug**
Hallucination metric score is being set to either 0.0 or 1.0. It never sets a score to a value in between of 0.0 and 1.0

**To Reproduce**
Steps to reproduce the behavior:
1. Create test cases where `actual_output` overlaps with `context`
2. Use hallucination metric to measure it
3. Run evaluation
4. Check hallucination score

**Expected behavior**
The hallucination score value is a continuous spectrum of real values <0.0; 1.0>

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS] Windows 11
 - Browser [e.g. chrome, safari] Chrome
 - Version [e.g. 22] 122

**Smartphone (please complete the following information):**
 - Device: [e.g. iPhone6]
 - OS: [e.g. iOS8.1]
 - Browser [e.g. stock browser, safari]
 - Version [e.g. 22]

**Additional context**
This problem might be connected with another one (the same test can get 0.0 in one run and 1.0 in another run, despite having the same input values)

Can you provide an example test case that has this behvaior?
Repository: confident-ai/deepeval
Issue: Hallucination metric score is set to 0.0 in one run and 1.0 in another run, despite having the same input values
Root cause: Algorithmic Error

**Describe the bug**
Hallucination metric seems to have low repeatability, in terms of what values it assigns to the same input values

**To Reproduce**
Steps to reproduce the behavior:
1. Create test cases where `actual_output` overlaps with `context`
2. Use hallucination metric to measure it
3. Run evaluation for a several times, for the same model output
4. Check hallucination scores

**Expected behavior**
The hallucination score is consistently the same or at least similar.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS] Windows 11
 - Browser [e.g. chrome, safari] Chrome
 - Version [e.g. 22] 122

**Smartphone (please complete the following information):**
 - Device: [e.g. iPhone6]
 - OS: [e.g. iOS8.1]
 - Browser [e.g. stock browser, safari]
 - Version [e.g. 22]

**Additional context**
This problem might be connected with another one (only either 0.0 or 1.0 are assigned to hallucination metric score, instead of using a full continuous space of numbers)

Can you provide an example test case that has this behvaior?
Repository: confident-ai/deepeval
Issue: Use Poetry build instead of setup.py
Root cause: Configuration Error

There is a discrepancy between the version of some of the libraries you use in setup.py vs the ones in pyproject.toml
Solution;
Use poetry build when generatin new release of the lib

Also can you accept Python3.12 version ?

@FrancoisMasson1990 python 3.12 ok in latest release :)
Repository: confident-ai/deepeval
Issue: Number of golden records in output is not the same as requested in input during synthesis
Root cause: Documentation Deficiency

Hi, I am facing an issue with response of golden records of synthesizer.
Even if I put max_goldens_per_document=1, it still returns 30+ records in response

```
params = {
            "synthesizer": self.synthesizer,
            "document_paths": ['path/to/document.pdf'],
            "max_goldens_per_document": 1,
            "include_expected_output": True,
            "chunk_size": 1024,
            "chunk_overlap": 0,
            "num_evolutions": 3
        }
        filtered_params = {k: v for k, v in params.items() if v is not None}
        dataset = EvaluationDataset()
        dataset.generate_goldens_from_docs(**filtered_params)
```

Hey @nalin-programmer, the docs from generate from context is outdated unforunately so actually the max goldens per document parameter isn't really doing anything, will show the updated docs soon but in the meantime we recommend using the method outside of the dataset and do it separately via the synthesizer: https://docs.confident-ai.com/docs/synthesizer-generate-from-docs#generate-your-goldens

Thanks for the help 

@nalin-programmer no problem
Repository: confident-ai/deepeval
Issue: Make TELEMETRY_DATA_FILE located into temp folder or configurable by env var
Root cause: Architectural Constraint

**Is your feature request related to a problem? Please describe.**
When running containerized apps in production the containers can't have filesystem set as road-only. 

**Describe the solution you'd like**
Would be great to set default path for TELEMETRY_DATA_FILE  to be in temp folder, which is mostly writable even if the rest of the FS is read-only. Or best let it define a developer using env var.

**Describe alternatives you've considered**
I can let virtualenv writable, but it's not good.

**Additional context**
Please think about security issues and do not write file wherever it drops.

Thank you
Repository: confident-ai/deepeval
Issue: copy_metrics from deepeval.metrics.utils does not copy arguments when they're passed via `**kwargs` instead of named arguments.
Root cause: Algorithmic Error

**Describe the bug**
A clear and concise description of what the bug is.

I have implemented a custom correctness metric as a class which inherits from `GEval`. Its constructor is similar to the following:
```python
class CustomCorrectness(GEval):
    def __init__(self, **kwargs):
        config_location = kwargs.get('config_location', None)
        # I do things here
        super().__init__(
        **kwargs
        )
```
I pass a custom LLM via `kwargs` which gets passed to `GEval`. When running `deepeval.evaluate()` with `run_async = False`, the custom LLM is used. However, when `run_async = True`, the evaluation happens with `gpt-4o`. This behaviour seems unintuitive, hence I think it's a bug, but I may be wrong.

I have tracked this down to the functionality of `copy_metrics()` from `deepeval.metrics.utils`. Since the class takes everything via `kwargs`, the `valid_params` end up being `odict_keys(['self', 'kwargs'])`, and hence the `model` passed via `kwargs` does not reach `GEval`, which defaults to `gpt-4o` (the default).

The simplest solution would be for me to implement the class and name all arguments, i.e. change the constructor:
```python
class CustomCorrectness(GEval):
    def __init__(self, model = None, config_location = 'location.yaml'):
        # I do things here
        super().__init__(
            model = model,
        )
```

However, the `copy_metrics` class could include some warning that behaviour may be unexpected when `'kwargs' in valid_params` so that future generations save time.

I have also played around with modifying the construction of `valid_params` where it iterates through `metric_class.__mro__`, collecting the signatures and using those as `valid_params`. Then it copies all of the arguments required for the superclasses.

I've tried implementing the solution, will soon create a pull request.

Please be patient, it's my first time contributing to open source projects, so this is a learning experience for me. The tests and the solutions may be clunky. I created a test in my branch in `tests/test_copy_metrics.py`, and it seems to reproduce the behaviour.

**To Reproduce**
The test in `tests/test_copy_metrics.py` reproduces the bug.

**Expected behavior**
Either a correct copying or a warning to not use `kwargs` in custom metrics.


**Desktop (please complete the following information):**
 - OS: Ubuntu 24.04.1 LTS


Also if there's any advice for contributing - please, I'd love to contribute more :)

Closing since it got merged :)
https://github.com/confident-ai/deepeval/pull/1202#issuecomment-2513718875
Repository: confident-ai/deepeval
Issue: ContextualPrecision endless generation
Root cause: Validation Gap

**Describe the bug**
For longer/more complex contexts, my evaluator stalls in the `ContextualPrecision._generate_verdicts()` in an endless generation loop. I can reproduce this in an LLM playground but unfortunately I can't share the exact contexts here.

I'm 'only' using llama 3.1 70B as evaluator, so using a more capable model would probably help, but I was actually able to resolve this by simply adding the number of nodes to the prompt *(this has helped me before with similar issues)*. 

I'm happy to submit a PR containing the change, then you can decide whether to integrate or not. Are you interested?

Hey @enrico-stauss I would be interested - but before you do anything do you know why it stalls in an endless loop?

Hi @penguine-ip , I believe it is the phenomenon that is also documented here: https://www.reddit.com/r/LocalLLaMA/comments/1ap8mxh/what_causes_llms_to_fall_into_repetitions_while/

I had observed similar issues before when I just provided a relatively long/complex prompt into the Groq playground - this time it happened on Azure Foundry. The framework I'm using for the LLM API is LlamaIndex by the way. The prompt itself is formatted through the `ContextualPrecisionTemplate.generate_verdicts` template and the contexts are 10 nodes of which some contain relatively complex html tables. 
I could probably demonstrate it to you if we would set up a meeting.

I think the reason that it enters an endless loop might be that in a long list of very long contexts the LLM fails to recognize how many sources actually are provided which is also why it can be fixed easily by just explicitly providing the true number of sources in the prompt.

Hey @enrico-stauss i see that's very helpful... I thought it was something wrong with deepeval. Yes would love a fix, could you please put out a PR? Really appreciate it!
Repository: confident-ai/deepeval
Issue: ModuleNotFoundError: No module named 'nest_asyncio'
Root cause: Environment Incompatibility

**Describe the bug**
I encountered a `ModuleNotFoundError: No module named 'nest_asyncio'` error after installing the `deepeval` library. It seems that `nest_asyncio`, which is required by the library, is not automatically installed as a dependency. 

**To Reproduce**
Create a new virtual environment:
```bash
python3 -m venv .venv  
source .venv/bin/activate 
```
Install uv & deepeval
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv init
uv add deepeval
```
Run a python file with a deepeval import (`python3 main.py`)
```python
import deepeval
```

**Potential Cause:**
I'm no python expert, but the `nest_asyncio` package is not included in the `install_requires` section of `setup.py`. Might this be the root cause of the issue, as `nest_asyncio` is required by `deepeval` but not automatically installed unless explicitly listed in `install_requires`.

**Additional context**
I'm using UV as a package manager and python 3.12.3.
Repository: confident-ai/deepeval
Issue: Commented piece of code for Golden creation
Root cause: Validation Gap

Hi,
It looks to me as an issue.
Or is there any reason why this `context_quality` metric value is excluded when creating golden in async way while it is considered for the sync one?
https://github.com/confident-ai/deepeval/blob/dd11c22b93d008ce0e386c88c86c96a38ad7c4f4/deepeval/synthesizer/synthesizer.py#L474

```python
            # Synthesize Golden
            golden = Golden(
                input=evolved_input,
                context=context,
                expected_output=expected_output,
                source_file=(
                    source_files[index] if source_files is not None else None
                ),
                additional_metadata={
                    "evolutions": evolutions_used,
                    "synthetic_input_quality": scores[i],
                    # "context_quality": (
                    #     context_scores[i]
                    #     if context_scores is not None
                    #     else None
                    # ),
                },
            )
            goldens.append(golden)
```

@kritinv can provide more info on this?
Repository: confident-ai/deepeval
Issue: Documentation for ContextualPrecision
Root cause: Documentation Deficiency

The [documentation ](https://docs.confident-ai.com/docs/metrics-contextual-precision) states that contextual precision uses the `actual_output` which is not true as it can be seen [here](https://github.com/confident-ai/deepeval/blob/dd11c22b93d008ce0e386c88c86c96a38ad7c4f4/deepeval/metrics/contextual_precision/contextual_precision.py#L66-L74).
Repository: confident-ai/deepeval
Issue: Metric for  Answer Precision/Completeness
Root cause: Documentation Deficiency

# Answer Precision/Completeness
I was going through the existing metrics and realized that there does not seem to be one that measures whether the `actual_answer` contains all information as per the `expected_answer` and maybe how similar it is in terms of detail or extent.

As this usecase does not sound very niche to me, I'd expect that this has been considered before. If that is the case, can someone please point me to a discission on the subject?
If there is no good reason, would you be interested in a PR that adds such metric to deepeval?

Hey @enrico-stauss often times for a simple comparison a GEval metric will be enough. Simply ask it to compare the actual output and expected output and it will work. The advantage of this is it leaves the definition of what is considered "information" up to the user to define. But of course it involves a few extra minutes of setup, let me know if it doesn't work for you?

GEval docs: https://docs.confident-ai.com/docs/metrics-llm-evals#example

@penguine-ip  you're right that solves it for me. Thanks!