zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Results are not in the same order as input test cases
Root cause: Algorithmic Error

Evaluation results are out of order, which makes it a pain to use them in a downstream process, or if I'm evaluating 2 models predictions at once. The 2 model's predictions and scores get mixed up.

**Steps to reproduce the behavior**:

```
from deepeval import evaluate
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

cases = []
for idx in range(3):
    cases.append(LLMTestCase(
        input=f"input{idx}",
        actual_output=f'The context says speaker {idx} is a bad person',
        retrieval_context=[f"context{idx}"])
    )

metric = FaithfulnessMetric(model='gpt-4o',include_reason=True)

results = evaluate(cases, [metric], print_results=False)

for idx,i in enumerate(results):
    print(i.input)
    print(i.actual_output)
    print(i.retrieval_context)
    print(cases[idx].retrieval_context)
    print(i.metrics_data[0].verbose_logs)
    print('\n\n')
```

**Expected behavior**
The results list should be ordered the same way as cases list.



I encountered the same issue. I am trying to match the metadata of the test cases in the evaluation dataset with the test results. Without the preservation of order, this is impossible..

Take a look at the `name` attribute of the `TestResult` class instance. It follows the format `test_case_i`, where `i` represents the position of the test case in the order it was passed to `evaluate()`. By extracting this number, you can rearrange the samples to match their original order.

I think the best would be, when loading the dataset with methods such as `add_test_cases_from_json_file`, to have a parameter `id_key` to be used as the test case name. 
Today the workaround is if you have multiple cases in a file, to parse it on your own and set the `LLMTestCase`. But this is a pity it should be a functionality when loading the datasets.
Repository: confident-ai/deepeval
Issue: ChromaDB dependency missing
Root cause: Environment Incompatibility

1. Follow this guide: https://docs.confident-ai.com/docs/evaluation-datasets-synthetic-data
2. Run `synthesizer.generate_goldens_from_docs()`
3. Get this error:

```
from chromadb.api.models.Collection import Collection
ModuleNotFoundError: No module named 'chromadb'
```

docs updated here: https://docs.confident-ai.com/docs/evaluation-datasets-synthetic-data#1-generating-from-documents, thanks! @fabiannagel
Repository: confident-ai/deepeval
Issue: Unable to create API keys, website reloading again and again.
Root cause: Others

**Describe the bug**
I am unable to create API keys for confident-AI when trying out a free demo. https://app.confident-ai.com/ is reloading for no reason.

**Expected behavior**
Normally making API keys because, it offers 2 free projects as the free quota as stated in pricing.

**Screenshots**
![image](https://github.com/user-attachments/assets/32f8d5cc-489c-46ad-ac96-42036c426ed3)
![image](https://github.com/user-attachments/assets/f3703e1b-9dfd-4b0d-a379-45033604a26a)

**Desktop (please complete the following information):**
 - OS: Windows 11
 - Browser: Brave, Chrome


Hey @aarushiksk what is the email you logged in with? You can email support@confident-ai.com to share your details
Repository: confident-ai/deepeval
Issue: How does one crete an account on https://www.confident-ai.com/?

On pricing page (https://www.confident-ai.com/pricing) it says there's a free version you can use for testing.

"no credit card required*" yet you can't access anything unless you enter your credit card.
Then if you downgrade to free account you lose access as all you can see is a message with "No access to Confident AI".

I'm sorry if this was the wrong place to ask this question, but not sure where else to do it as there's no contact form on the site. 

You can email ***@***.***, and to answer your question the
website is just outdated by one sentence. When you signed up it clearly
says start a 7 day free trial and you can cancel anytime, which it asked
for your card details, which is still free. When you downgrade it tells you
you will lose access so you should use your 7 days to decide if you like it
or not, I'm guessing you didn't read it and are confused for this reason.

On Wed, Oct 2, 2024 at 6:41 PM Daniel Anechitoaie ***@***.***>
wrote:

> I'm sorry if this was the wrong place to ask this question, but not sure
> where else to do it as there's no contact form on the site.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/confident-ai/deepeval/issues/1044#issuecomment-2388330871>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BCFQK62RI3EJTEUX4AVD3WDZZPEUHAVCNFSM6AAAAABPF2V5BWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBYGMZTAOBXGE>
> .
> You are receiving this because you modified the open/close state.Message
> ID: ***@***.***>
>


Ok. It's clear now. Thanks.
Repository: confident-ai/deepeval
Issue: can we disable telemetry by default in python client? 
Root cause: Configuration Error

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. add deepeval in any project, and start the service it without setting DEEPEVAL_TELEMETRY_OPT_OUT = YES

**Expected behavior**
A clear and concise description of what you expected to happen.

telemetry should be disabled, otherwise deepeval gets a lot of private information from application

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Smartphone (please complete the following information):**
 - Device: [e.g. iPhone6]
 - OS: [e.g. iOS8.1]
 - Browser [e.g. stock browser, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.


@pritamsoni-hsr hey not at the moment, happy to merge a quick PR on this

hey @penguine-ip opened a PR for the fix, the CI seems to be failing tho

```
Did not find openai_api_key, please add an environment variable `OPENAI_API_KEY` which contains it, or pass `openai_api_key` as a named parameter. (type=value_error)
```

Hey @pritamsoni-hsr we would accept a PR where there is an option to set opt out in python like
```
deepeval.disable_telemtry()
```

The current way of default opt out means we won't get feedback on what features are most popular.
Repository: confident-ai/deepeval
Issue: Support function calling fields `expected_tools` and `tools_called` in`Golden`
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
The `Golden` object should support `expected_tools` and `tools_called` (similar to `LLMTestCase`) for Tool Correctness / Function Calling.

**Describe the solution you'd like**
Add `expected_tools` and `tools_called` as fields into `Golden`

**Describe alternatives you've considered**
Adding these fields as `additional_metadata` is an option but is consistent with the existing use-cases as well as the definition of `LLMTestCase`

**Additional context**
Add any other context or screenshots about the feature request here.


Working on a PR rn
Repository: confident-ai/deepeval
Issue: Update `load_benchmark_dataset` to ensure correct typings for method implementations
Root cause: Interface Contract Mismatch

**Is your feature request related to a problem? Please describe.**
`DeepEvalBaseBenchmark.load_benchmark_dataset()` typing does not match the expected abstract method implementations. Specifically, `load_benchmark_dataset()` should be able to take in some arbitrary number of parameters like the child class `TruthfulQA` where

```py
def load_benchmark_dataset(self, task: TruthfulQATask, mode: TruthfulQAMode) -> List[Golden]
```

**Describe the solution you'd like**
Change from
```python
class DeepEvalBaseBenchmark(ABC, Generic[T]):
    def __init__(self, dataset: Optional[Dataset] = None):
        self.tasks: List[T] = []
        self.dataset = dataset

    @abstractmethod
    def load_benchmark_dataset(self) -> List[Golden]:
        """Load the benchmark dataset and initialize tasks."""
        raise NotImplementedError
```

```python
    ...

    @abstractmethod
    def load_benchmark_dataset(self, *args, **kwargs) -> List[Golden]:
        ...
```

**Describe alternatives you've considered**
n/a

**Additional context**
Repository: confident-ai/deepeval
Issue: Why are the scorer tests commented out?
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
> A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Hi. Just curious, why are the tests at https://github.com/confident-ai/deepeval/blob/main/tests/test_scoring.py commented out? I'd like to build a custom metric based on the scorer functions in https://github.com/confident-ai/deepeval/blob/main/deepeval/scorer/scorer.py but not sure if the tests being commented indicate that any of the scorer functions have bugs in their logic. 

The [last PR that touches the former file](https://github.com/confident-ai/deepeval/pull/493) dates back to Feb 21 2024. 

**Describe the solution you'd like**
> A clear and concise description of what you want to happen.

Perhaps uncommented tests and/or improved/additional official documentation at https://docs.confident-ai.com/docs/metrics-custom on why these tests are commented. 


@jtquach1 Hey it was commented out because the scorers while used in the much earlier versions of deepeval are no longer used. The unit tests are mainly for internal testing on our end so we comment out the ones that we know won't be affecting the package
Repository: confident-ai/deepeval
Issue: Question About The Syntesizer 

https://www.confident-ai.com/blog/the-definitive-guide-to-synthetic-data-generation-using-llms
Why it separates the query generation from the label (answer) generation?

My initial guess would bee that generating the question with the answer will be more coherent than building questions and then completely separately sending the question to generate an answer. I would say that even from the cost perspective it makes less sense. (doubling the calls and the amount of tokens ) . So I assume they there is good reason about it?
Repository: confident-ai/deepeval
Issue: Add a "not sure" / "unwilling to answer" metric
Root cause: Unimplemented Feature Gap

Measure the amount of times an LLM is "unsure" of something or "unwilling" to answer. This is a growing pain point of LLMs. Not sure if research papers have caught up in this area unfortunately so some sort of Conceptual Similarity across a few such prompts should be the easiest way to do this. 

Good call on this - this would be a subset of 'answer relevancy', correct?  If so, any thoughts on some kind of an integration between the 2 or is the 'not sure' metric envisioned to be seperate?
https://github.com/confident-ai/deepeval/blob/main/tests/test_answer_relevancy.py 

Yep that's a really really good point. Currently - I see it as 2 separate metrics and but what we can do is add an overarching "Answer Relevancy" metric that then shows a few ways to measure "Answer Relevancy".
Repository: confident-ai/deepeval
Issue: Missing max_goldens_per_document argument in synthesizer.generate_goldens_from_docs
Root cause: Interface Contract Mismatch

Hello,

First of all, thank you for maintaining this amazing project. I’ve been using it recently and came across a potential issue.

In the function generate_goldens_from_docs（deepeval/deepeval/dataset/dataset.py）, it seems that the synthesizer's generate_goldens_from_docs method is called with a max_goldens_per_document argument. However, based on my investigation, I could not find max_goldens_per_document as an argument in the synthesizer's implementation of generate_goldens_from_docs.

https://github.com/confident-ai/deepeval/blob/8d0e8346f9f506a9833abd691be3b735c6651e19/deepeval/dataset/dataset.py#L527


I wanted to check if this argument should be supported or if it's an oversight in the documentation or code.

Thank you for your time and support! I'd appreciate any clarification on this.



I second this, currently, the Synthesizer supports the following two related parameters:
- `max_goldens_per_context`
- `max_contexts_per_document`

@NaraSyuhei @adesso-dominik-chodounsky thanks so much! out in v1.3.6
Repository: confident-ai/deepeval
Issue: Upon failing a certain test, deepeval could print the LLM's output
Root cause: Unimplemented Feature Gap

I hope I did not overlook something, but I could not find/figure out how to get deepeval's `assert_test` function to print the evaluated LLM's answer in the shown report.

The report shows that for the given test case and metric, the test failed and gives the reason why, but it would be nice to be able to specify that the LLM's output should be included in the report for debugging reasons. Perhaps context too, I would suggest having the two separate, as the context may often be quite long.

@adesso-dominik-chodounsky Hey! Its not possible to print everything in a user friendly way since its just the terminal we're writing to. Do you have some other methods in mind?
Repository: confident-ai/deepeval
Issue: dataset.generated_goldens buggy because of wrong type of evolutions
Root cause: Interface Contract Mismatch

While trying to use EvaluationDataset().generate_goldens the following error appeared: 

```
AttributeError: 'list' object has no attribute 'keys'
in Synthesizer._a_evolve_input(self, input, num_evolutions, evolutions, context, progress_bar)
    883 evolutions_used = []
    884 for _ in range(num_evolutions):
    885     # Randomize Evolution
    886     evolution_type = random.choices(
--> 887         list(evolutions.keys()), list(evolutions.values())
    888     )[0]
    890     # Create Evolution Prompt
    891     if isinstance(evolution_type, Evolution):
```

This is beacuse the default evolutions of the EvaluationDataset().generate_goldens is a list, while the needed object for synthesizer.generate_goldens() is a dictionary with the fractions. 

If I have analyzed this error correctly, please change the Default type of generate_goldens in the evaluation dataset. Thank you!



Any update on this issue please ? I had the same problem. 

I'm sorry, I hadn't read this issue before and ended up creating a duplicate issue. I believe the root cause is related to what I mentioned in my issue: the default values for the evolutions argument seem to be incorrect.

https://github.com/confident-ai/deepeval/issues/1085
Repository: confident-ai/deepeval
Issue: Wrong interpretation of the result of Hallucination
Root cause: Algorithmic Error

I understand that if it is a "yes", there is alignement with context, and if there is "no", there is contradiction. However when you assign the verdicts to the list "contradictions" or "factualalignements", you add the verdicts with a "no" inside the factualalignements instead of contradiction

https://github.com/confident-ai/deepeval/blob/3ce770c2a512d64c84d2aa07dd2e3879beda2bbc/deepeval/metrics/hallucination/hallucination.py#L119

Hey @anotine10 totally, thanks for bringing this up, changing it right now :)

@penguine-ip is this closed or do we need help here?


Hey @sahibpreetsingh12  its closed and out in the latest release, appreciate it!
Repository: confident-ai/deepeval
Issue: evolutions argument should be of type Dict[Evolution, float] instead of List[Evolution]
Root cause: Interface Contract Mismatch

Hello,

Thank you once again for your ongoing improvements to this project. I've been using it actively and noticed another potential issue.

In the current implementation of the generate_goldens_from_docs function, the evolutions argument is defined as List[Evolution]. However, it seems that the correct type should be Dict[Evolution, float], as each evolution might require a weight or priority for generating goldens.

Here is the relevant portion of the code:

https://github.com/confident-ai/deepeval/blob/dd64174925ccbacd4bb1feb332a9dcce29d87bc3/deepeval/dataset/dataset.py#L511

I believe allowing the evolutions argument to accept weights would offer more flexibility, enabling users to prioritize evolutions. Could you please confirm if this adjustment would be appropriate?

Thank you again for your hard work, and I look forward to your feedback.

@NaraSyuhei Thanks for bringing this up! Will change the typing to avoid any confusion :)
Repository: confident-ai/deepeval
Issue: Missing API key and endpoint configuration...(etc) in OpenAI() and support for Azure OpenAI
Root cause: Documentation Deficiency

Hi

Thank you for your continued efforts on this project. I’ve encountered an issue while using the following code:

https://github.com/confident-ai/deepeval/blob/dd64174925ccbacd4bb1feb332a9dcce29d87bc3/deepeval/models/gpt_model_schematic.py#L114

https://github.com/confident-ai/deepeval/blob/dd64174925ccbacd4bb1feb332a9dcce29d87bc3/deepeval/models/gpt_model_schematic.py#L132

It seems that the OpenAI() initialization is missing some necessary configurations, such as the API key and endpoint settings. Without these configurations, it doesn't seem to function as expected.

Additionally, I would like to suggest making it compatible with Azure OpenAI as well. This would broaden the utility of the library for those using Azure's services.

Could you please confirm if the OpenAI() configuration needs to include these parameters, and if there's a possibility to support Azure OpenAI in the future?

Thank you for considering this, and I appreciate your hard work!

Hey @NaraSyuhei, have you tried setting the openai api key in the env variables?

Azure OpenAI is already supported, you can also implement your own wrappers: https://docs.confident-ai.com/docs/guides-using-custom-llms

@penguine-ip  On a similar issue, can I use custom LLM for the synthesizer please?

@yfchau-news definitely, also don't forget to include the custom model you want for the configs in the generation pipeline (filter config for example uses a model) https://docs.confident-ai.com/docs/synthesizer-introduction#create-your-first-synthesizer
Repository: confident-ai/deepeval
Issue: Typo in scorer/scorer.py

**Is your feature request related to a problem? Please describe.**

Typo in scorer.py

**Describe the solution you'd like**

The string on line 78 says "Invalud", but I think it should say "Invalid". [Source code](https://github.com/confident-ai/deepeval/blob/dd64174925ccbacd4bb1feb332a9dcce29d87bc3/deepeval/scorer/scorer.py#L78). 

I can't make a PR so just noting this here.
Repository: confident-ai/deepeval
Issue: Incompatibility with ragas v0.2.x causing TypeError in measure function
Root cause: External Dependency Breakage

**Describe the bug**
I encountered an issue while using deepeval 1.3.9 with ragas v0.2.1.
The measure function throws a TypeError due to the change in the return format of the evaluate function in ragas v0.2.x. Here are the details:

```bash
  File "lib/python3.10/site-packages/deepeval/metrics/ragas.py", line 389, in measure
    score = metric.measure(test_case)
  File "lib/python3.10/site-packages/deepeval/metrics/ragas.py", line 269, in measure
    self.success = faithfulness_score >= self.threshold
TypeError: '>=' not supported between instances of 'list' and 'float'
```

**To Reproduce**
Steps to reproduce the behavior:
1. Install deepeval `1.3.9` and ragas `v0.2.1`.
2. Run the code involving measure function with RAGAS metric.
```py
from deepeval.test_case import LLMTestCase
from deepeval.metrics.ragas import RAGASFaithfulnessMetric

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

# Replace this with the expected output from your RAG generator
expected_output = "You are eligible for a 30 day full refund at no extra cost."

# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = ["All customers are eligible for a 30 day full refund at no extra cost."]

metric = RAGASFaithfulnessMetric(threshold=0.5, model="gpt-3.5-turbo")
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output,
    expected_output=expected_output,
    retrieval_context=retrieval_context
)

metric.measure(test_case)
print(metric.score)
```
3. The error occurs when scores["faithfulness"] is a list, while it is expected to be a float.





**Expected behavior**
Could you please look into this and adjust deepeval for compatibility with the new return format of ragas v0.2.x? 
Alternatively, you could consider adding a version constraint for ragas in pyproject.toml to ensure users install a compatible version until this issue is resolved. Thank you!

**Desktop (please complete the following information):**
 - OS: Ubuntu
Repository: confident-ai/deepeval
Issue:  Accuracy Issue - accuracy is 0.00
Root cause: Configuration Error

When I evaluate the score based on the MMLU dataset, the overall accuracy is 0.00, and the predicted answers do not match the actual answers. How can I modify the prompt to generate responses that include the index of the choices?

I have the same issue here. I think the prompt is not passed into the model correctly. As MMLU dataset required a specific prompt strategy
Repository: confident-ai/deepeval
Issue: Litellm integration request
Root cause: Unimplemented Feature Gap

I think it's a good idea to integrate with the Litellm as it provides an easy to integrate with 100+ LLMs.

https://github.com/BerriAI/litellm

LLM as a drop in replacement for GPT. Use Azure, OpenAI, Cohere, Anthropic, Ollama, VLLM, Sagemaker, HuggingFace, Replicate (100+ LLMs)

Hey - yeah! We're planning to integrate with them very soon :) 

Update on this - I'll be writing a guide on how to use Litellm with us instead of a full-blown integration. 
Should just be a few lines of code! 

https://litellm-git-fork-colabdog-feature-add-deepeval-clerkieai.vercel.app/docs/tutorials/deepeval