zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: data.generate_goldens throw TypeError: Synthesizer.generate_goldens() got an unexpected keyword argument 'enable_breadth_evolve'
Root cause: Interface Contract Mismatch

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
When i run dataset.generate_goldens() follow document i got error
TypeError: Synthesizer.generate_goldens() got an unexpected keyword argument 'enable_breadth_evolve'
If i investigate deepeval code. `Dataset.generate_goldens` pass  `enable_breadth_evolve` which is not a keyword of `Synthesizer`


**To Reproduce**
Steps to reproduce the behavior:
SImply Run dataset.generate_goldens() you got error.
TypeError: Synthesizer.generate_goldens() got an unexpected keyword argument 'enable_breadth_evolve'


**Expected behavior**
No error.


**Screenshots**
<img width="769" alt="스크린샷 2024-07-25 오후 10 59 01" src="https://github.com/user-attachments/assets/18cfd912-de44-4078-9935-23716ce1c52b">
<img width="692" alt="스크린샷 2024-07-25 오후 10 59 11" src="https://github.com/user-attachments/assets/55a33188-129c-422a-907e-d3c0ecf70c5e">



**Desktop (please complete the following information):**
 - OS: OSX
 - Browser chrome
 - Version 126



**Additional context**
deepeval  version: 0.21.68


@gring2 I believe it is fixed in the newest version v0.21.74, can you let me know if you're still running into errors? Thanks!

> @gring2 I believe it is fixed in the newest version v0.21.74, can you let me know if you're still running into errors? Thanks!

I checked it is fixed. Thank you for your work
Repository: confident-ai/deepeval
Issue: Suggestion to modify the "valid_gpt_models" list from gpt_model.py
Root cause: Architectural Constraint

Thank you for your helpful library.

I am writing to suggest modifying the [valid_gpt_models](https://github.com/confident-ai/deepeval/blob/36911d498edad3c15997f18c0011a52bca284f89/deepeval/models/gpt_model.py#L20C1-L20C17) list.

The model pointing provided by OpenAI can occasionally change, but the DeepEval code cannot always be updated simultaneously. For example, "gpt-4o" currently points to "gpt-4o-2024-05-13" (as of July 2024), but it will be changed to "gpt-4o-2024-08-06" soon. This variability can lead to inconsistencies and confusion in the evaluation results, affecting the reliability and consistency of DeepEval's assessments.

Therefore, I suggest replacing the hard-coded list with the following code:
```python
client = openai.OpenAI()
valid_gpt_models = [openai_model.id for openai_model in client.models.list().data]
```

Thank you for considering this suggestion.

I hope this helps!
Repository: confident-ai/deepeval
Issue: Why actual_output is required for Context Relevancy?

hi, thanks for building this great tool. i have been using this tool for a while, and found **actual_output** is actually not used by Contextual Relevancy metric, but the design is(from [docs](https://docs.confident-ai.com/docs/evaluation-test-cases)) **actual_output** is always needed.

@isaactl Thanks! Intentional since evaluation is owned by an LLMTestCase, which by default requires input and actual output regardless of whether they are used in evaluation. (wanted to make this a strictly LLM evaluation framework rather than any NLP evaluation task)

If you really don't want to provide an actual output, simply provide an empty string and it won't affect anything.
Repository: confident-ai/deepeval
Issue: Add jsonl file support in dataset 
Root cause: Unimplemented Feature Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.


@princepride What;s the issue here?

I noticed that your dataset includes loading data via JSON or CSV, could you add support for the jsonl file format?
Repository: confident-ai/deepeval
Issue: Recursion Error While Verbose_mode is set to True
Root cause: Algorithmic Error

I am getting the error when trying to use AnswerRelevancyMetric, ToxicityMetric alongwith with verbose_mode set to True. It is going in infinite recursion and throwing error :
RecursionError                            Traceback (most recent call last)
[/usr/lib/python3.10/threading.py](https://localhost:8080/#) in invoke_excepthook(thread)
   1324             args = ExceptHookArgs([*sys_exc_info(), thread])
   1325 
-> 1326             hook(args)
   1327         except Exception as exc:
   1328             exc.__suppress_context__ = True

11 frames
... last 10 frames repeated, from the frame below ...

[/usr/local/lib/python3.10/dist-packages/rich/console.py](https://localhost:8080/#) in __exit__(self, exc_type, exc_value, traceback)
    863     def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
    864         """Exit buffer context."""
--> 865         self._exit_buffer()
    866 
    867     def begin_capture(self) -> None:

RecursionError: maximum recursion depth exceeded while calling a Python object
---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/rich/console.py](https://localhost:8080/#) in print(self, sep, end, style, justify, overflow, no_wrap, emoji, markup, highlight, width, height, crop, soft_wrap, new_line_start, *objects)
   1673         with self:
-> 1674             renderables = self._collect_renderables(
   1675                 objects,

29 frames
RecursionError: maximum recursion depth exceeded in comparison

During handling of the above exception, another exception occurred:

RecursionError                            Traceback (most recent call last)
... last 10 frames repeated, from the frame below ...

[/usr/local/lib/python3.10/dist-packages/rich/console.py](https://localhost:8080/#) in __exit__(self, exc_type, exc_value, traceback)
    863     def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
    864         """Exit buffer context."""
--> 865         self._exit_buffer()
    866 
    867     def begin_capture(self) -> None:

RecursionError: maximum recursion depth exceeded while calling a Python object

I am using deepeval library version: 0.21.74

Same issue, when I create metrics with verbose_mode=True ; There occurs infinite loop
RecursionError: maximum recursion depth exceeded

The issue still persists..

<img width="582" alt="Image" src="https://github.com/user-attachments/assets/2d22a43e-a089-4989-8151-bfac6b6a2007" />
Repository: confident-ai/deepeval
Issue: Tool Correctness documentation page shows using ContextualRelevancyMetric instead of ToolCorrectnessMetric 
Root cause: Documentation Deficiency

**Describe the bug**
Tool Correctness documentation page shows using ContextualRelevancyMetric instead of ToolCorrectnessMetric. The example is incorrect.

https://docs.confident-ai.com/docs/metrics-tool-correctness

**To Reproduce**
Steps to reproduce the behavior:
1. Go to 'https://docs.confident-ai.com/docs/metrics-tool-correctness'
2. Check the example

**Expected behavior**
The document needs to be correct to show the example to use ToolCorrectnessMetric
[Tool Correctness _ DeepEval - The Open-Source LLM Evaluation Framework.pdf](https://github.com/user-attachments/files/16557206/Tool.Correctness._.DeepEval.-.The.Open-Source.LLM.Evaluation.Framework.pdf)


@vensr thanks!
Repository: confident-ai/deepeval
Issue: Parameters mentioned in Conversation Relevancy Doc page is of Faithfulness

**Describe the bug**
The parameters mentioned in the conversational doc page is incorrect. It mentions about Faithfulness parameters for Conversation Relevancy.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to 'https://docs.confident-ai.com/docs/metrics-conversation-relevancy'
2. Check the parameters section.

**Expected behavior**
The document needs to be corrected to show the Conversation Relevancy parameters.
[Conversation Relevancy _ DeepEval - The Open-Source LLM Evaluation Framework.pdf](https://github.com/user-attachments/files/16558198/Conversation.Relevancy._.DeepEval.-.The.Open-Source.LLM.Evaluation.Framework.pdf)


Hey @vensr thanks for bringing this up, do you mean this link? https://docs.confident-ai.com/docs/metrics-conversation-relevancy#required-arguments

Yes the same link.

@vensr it is correct.
Repository: confident-ai/deepeval
Issue: Example Colab notebook is failing. No module named 'deepeval.metrics.conversation_relevancy
Root cause: Validation Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
Example [Colab notebook](https://colab.research.google.com/drive/1PPxYEBa6eu__LquGoFFJZkhYgWVYE6kh?usp=sharing) is failing. 

```python 
import deepeval
```

```
ModuleNotFoundError                       Traceback (most recent call last)
[<ipython-input-5-9be8d21cb7e2>](https://localhost:8080/#) in <cell line: 1>()
----> 1 from deepeval.metrics import AnswerRelevancyMetric
      2 
      3 answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.5)

7 frames
[/usr/local/lib/python3.10/dist-packages/deepeval/metrics/__init__.py](https://localhost:8080/#) in <module>
     13 from .knowledge_retention.knowledge_retention import KnowledgeRetentionMetric
     14 from .tool_correctness.tool_correctness import ToolCorrectnessMetric
---> 15 from .conversation_relevancy.conversation_relevancy import ConversationRelevancy
     16 from .conversation_completeness.conversation_completeness import (
     17     ConversationCompleteness,

ModuleNotFoundError: No module named 'deepeval.metrics.conversation_relevancy'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------
```

**To Reproduce**
Steps to reproduce the behavior:
1. Open the [Colab notebook](https://colab.research.google.com/drive/1PPxYEBa6eu__LquGoFFJZkhYgWVYE6kh?usp=sharing) mentioned in the README file
2. Run pip install cell and set the env variables
3. Try to import deepeval
4. See error

**Expected behavior**
Import deepeval should word

**Environment
 - OS: Google Colab



@arianpasquali Hey can you try the latest version v0.21.77?
Repository: confident-ai/deepeval
Issue: Incorrect pydantic schemas cause ValidationErrors when using a custom LLM for evaluation
Root cause: Validation Gap

**Describe the bug**
Incorrect pydantic schemas cause `pydantic_core._pydantic_core.ValidationError`s when using a custom LLM for evaluation, which is implemented with schema injection as described [here](https://docs.confident-ai.com/docs/guides-using-custom-llms#reimplementing-the-generate-and-a_generate-methods). 

The problem occurs for any `Verdicts` schema which has `reason : str` instead of `reason : Optional[str]`, given that the prompt requires the llm-judge to return no reason for 'yes' verdicts. Additionally, the `Steps` schema for GEval should be `steps: List[str]` where it is currently `steps: str`. 

**To Reproduce**
Problem with `Verdicts` schema:
1. Implement a custom LLM for evaluation as described in the link above using pydantic schemas, making the generate() function return `json.loads('{\n"verdicts": [\n{\n    "verdict": "yes",\n    "reason":null \n}\n]\n}')`
2. Call `model.generate()` with `schema=Verdicts`
3. See error: 
```
pydantic_core._pydantic_core.ValidationError: 1 validation error for Verdicts
reason
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
```

Problem with the `Steps` schema for GEval metrics:
1.  Implement a custom LLM for evaluation as described in the link above using pydantic schemas, making the generate() function return `json.loads('{\n"steps": [\n   "step1",\n  "step2"\n]\n}')`
2. Call `model.generate()` with `schema=Steps`
3. See error: 
```
pydantic_core._pydantic_core.ValidationError: 1 validation error for Steps
steps
  Input should be a valid string [type=string_type, input_value=['step1', 'step2'], input_type=list]
```
Repository: confident-ai/deepeval
Issue: Error on dataset.pull: TypeError: Api.send_request() got an unexpected keyword argument 'params'
Root cause: Interface Contract Mismatch

dataset.pull does not work because it is sending a params keyword argument to Api.send_request() that does not exist. Checked the source code and it indeed does not look like that keyword argument is present.

**Screenshots**
![image](https://github.com/user-attachments/assets/9be535a9-ef3e-4225-ba6e-ed99acb7a19f)

**Desktop (please complete the following information):**
 - OS: Ubuntu 22.04
 - Browser Chrome
 - Version 22.04


@bendominguez0111 thanks for bringing this up, fixed in latest release (v1.1.0), thanks!
Repository: confident-ai/deepeval
Issue: Synthesizer - generate from scratch returns only one golden
Root cause: Algorithmic Error

Hey, 
I think I found a bug I would like to report.
Because of an indentation typo, synchronous version of the `generate_goldens_from_scratch ` method will use only one initial golden, and then return with the evolved versions of only that.
https://github.com/confident-ai/deepeval/blob/095e37e5372be82d68be846e74b5deaa804ef116/deepeval/synthesizer/synthesizer.py#L562

Works for async version.

somewhat related:
if num_evolutions=0 then the progress message is: 
`✨ 🍰 ✨ You're generating up to 0 goldens using DeepEval's latest Synthesizer`
Which is not true, it will generate `num_initial_goldens` goldens only won't be evolve it with evolutions.


Should be fixed in the next synthesizer update soon.

This problem still exists. 
(deepeval 3.3.6)
Ty
Repository: confident-ai/deepeval
Issue: About 'node' in prompt of contextual_precision
Root cause: Documentation Deficiency

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
I noticed that the contextual_precision prompt mentioned 'The retrieval contexts is a list of JSON with three keys: `verdict`, `reason` (reason for the verdict) and `node`. ' However, when I actually used the contextual_precision, I did not observe the generated prompt containing the 'node' key. Why is that?

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Smartphone (please complete the following information):**
 - Device: [e.g. iPhone6]
 - OS: [e.g. iOS8.1]
 - Browser [e.g. stock browser, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.


@Ishiki-Iroha Must be a bug we forgot to remove in the prompts, thanks for bringing it up!
Repository: confident-ai/deepeval
Issue: deepeval and api key
Root cause: Documentation Deficiency


**Is your feature request related to a problem? Please describe.**
i want to use deepeval with my local model on ollama, but it seems not possibile

**Describe the solution you'd like**
i want the possibility to use my api key instead of the Open AI key , i want to maintain my data private



open your terminal and shell 
deepeval set-local-model --model-name=llama3:8b --base-url="http://localhost:11434/v1/" --api-key="ollama"

Duplicate:
- https://github.com/confident-ai/deepeval/issues/1007
Repository: confident-ai/deepeval
Issue: timeout mechanism
Root cause: Unimplemented Feature Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
I noticed that the current evaluation system doesn't provide a mechanism for timeout handling, and in practice, retrying LLM reasoning indefinitely isn't robust in production
Adding a configurable maximum number of attempts or a maximum timeout to the retry decorator should be better.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.


Hey @Ishiki-Iroha,

Thank you very much for raising this issue. We have been hard at work making stability and reliability improvements to deepeval. Here are some contributions that add a well-defined retry policy across all models and address request timeout concerns to prevent deadlocks and otherwise indefinite hangs:

- https://github.com/confident-ai/deepeval/pull/1941
- https://github.com/confident-ai/deepeval/pull/2047
- https://github.com/confident-ai/deepeval/pull/2068

and more stability fixes like this are on their way as part of the hacktoberfeast:

- https://github.com/confident-ai/deepeval/issues/2128
- https://github.com/confident-ai/deepeval/issues/2126
- https://github.com/confident-ai/deepeval/issues/2125

There is more to come so keep your deepeval install up to date.

Cheers!


PR to add http timeouts for sync flows
https://github.com/confident-ai/deepeval/pull/2146
Repository: confident-ai/deepeval
Issue: bug: Tools used not showing up in test report
Root cause: Configuration Error

**Describe the bug**
I am running a ToolCorrectnessMetric test. I am expecting one tool to be called. Below is my code:
```
def test_tool_get_syllabus_correctness_for_all_functions():
    config = get_cosmos_data()
    params = {conf["course_name"]: conf for conf in config}
    course = [course for course in params][0]
    params = params[course]

    function_registry = create_function_registry(
        pull_announcements=True,
        pull_groups=True,
        pull_syllabus=True,
        pull_discussions=True,
        pull_custom_files=True,
        pull_course_content=True,
    )

    question = "What does the syllabus say about the final?"
    messages = [{"role": "user", "content": question}]
    expected_tools = ["get_syllabus"]
    ai_response, tools_used, _ = run_conversation_custom_funcs(
        function_registry, params["prompt_template"], messages, question, params, "Chip"
    )

    metric = ToolCorrectnessMetric()
    test_case = LLMTestCase(
        input=question,
        actual_output=ai_response,
        tools_used=tools_used,
        expected_tools=expected_tools,
    )
    print(f"tools_used: {tools_used}")

    assert_test(test_case, metrics=[metric])
    
```

   The test passes successfully and my  one tool call was printed but the tools_used is missing in the report that opens up on confident-ai.com.

  ✨ You're running DeepEval's latest Tool Correctness Metric! (using None, strict=False, async_mode=True)... Done! (0.00s)
tools_used: ['get_syllabus']

**To Reproduce**
Steps to reproduce the behavior:
1. Execute a test that relies on tool correctness
2. Pass the test as tools_used within LLMTestCase
3. Test passes but tool used not in web UI.

**Expected behavior**
The tools used should be populated in the WebUI.

**Screenshots**
![image](https://github.com/user-attachments/assets/0668c15e-a9db-4dc4-a8b9-be7464726061)

**Desktop (please complete the following information):**
 - OS: Windows 11
 - Browser Firefox
 - Latest version of deepeval


**Additional context**
Its entirely possible I am doing something wrong on my end. 


@chip-davis hey, seems like a problem on our side, will try to fix it by tomorrow, thanks!

> @chip-davis hey, seems like a problem on our side, will try to fix it by tomorrow, thanks!

Sure! 

Hey @chip-davis it is fixed now! Might need the latest deepeval version tho (v1.1.4)
Repository: confident-ai/deepeval
Issue: deepeval AttributeError: 'str' object has no attribute 'answer'
Root cause: Documentation Deficiency

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Smartphone (please complete the following information):**
 - Device: [e.g. iPhone6]
 - OS: [e.g. iOS8.1]
 - Browser [e.g. stock browser, safari]
 - Version [e.g. 22]

**Additional context**

in class MMLU
Add any other context about the problem here.
            res: MultipleChoiceSchema = model.generate(
                prompt=prompt, schema=MultipleChoiceSchema
            )
            prediction = res.answer
        except TypeError:
            prompt += (
                "\n\nOutput 'A', 'B', 'C', or 'D'. Full answer not needed."
            )
            prediction = model.generate(prompt)


for res from model.generate there's no answer attribute  
and i already create a custom model from example like https://docs.confident-ai.com/docs/benchmarks-introduction



Hey @habout632. Please try implementing your custom model according to the following guide: https://docs.confident-ai.com/docs/guides-using-custom-llms#creating-a-custom-llm. If you pass a schema argument for your custom model, JSON enforcement should handle the answer attribute automatically. If no such argument is provided in the generate function signature, your model should trigger the TypeError section.

I did the same as the link there and get the same type of error with aws bedrock claude 3.5 sonnet v2. Is there anyway to know what the schema is the framework sends so its easier to ID what the root cause is?