zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Having so many random errors while scoring
Root cause: Validation Gap

**Describe the bug**
In order to design a validation pipeline, I started to use deepeval. First, thanks for the package, it's the only app I saw that supports the open-source models as an evaluation model.

But, I face many random errors for the metrics below:
- AnswerRelevancyMetric
- ContextualPrecisionMetric
- ContextualRelevancyMetric
- ContextualRecallMetric

As a custom model,  I use `TheBloke/Mistral-7B-Instruct-v0.2-GGUF` model with the llama.cpp.

The errors I see:
- ValueError: Number of verdicts generated does not equal length of retrieved nodes
    - I checked the other issue and tried different `n_context` values but it seems it didn't work.
- JsonEncoder issues (for example: Expecting ',' delimiter: line 10 column 10)

As a workaround, I do try-except with retry. Do you have any other suggestions or ideas as to why it happens?

Besides, I would like to ask one more question. In order to have more consistent results, do we need to have multiple scores and use the average of these scores?

Regards

**Desktop:**
 - OS: MacOS 14.2.1
 - Browser: Chrome
 - Version: 0.20.66


@ktoprakucar It seems like your model isn't good enough for evaluation. Basically:

"Evaluations may not always work as expected when using a custom model. This is because evaluation requires: 1) high levels of reasoning that are generally hard to achieve with custom models, 2) the ability to follow instructions such as outputting responses in JSON formats."

So the first error you're encountering is from the Contextual metrics which tries to determine whether each retrieval context is relevant to the input. When an LLM isn't good enough, its not able to say whether EACH context is relevant, and so the subsequent calculation for precision and recall doesn't make sense, hence we return an error instead.

For the second error, your model isn't outputting valid JSON as instructed. Is it a custom model you must use? Would love to learn more about your use case to see any workarounds

Hey @penguine-ip ,

Thanks for your detailed answer! 

Unfortunately, I need to use custom models because of policies. Do you have any other suggestion/model that can be capable of the French language, it would be more than welcome! :)

@ktoprakucar I would assume mistral, but I have had reports of it not consistently outputting JSON...

@penguine-ip Thanks for the suggestion! 

We already tested Mistral 7B (mistral-7b-instruct-v0.2.Q6_K.gguf) and Mixtral 8X7B (mixtral-8x7b-instruct-v0.1.Q3_K_M.gguf) and had the same errors. 
We are planning to upgrade our Hardware to test more complex models. I plan to use your library with more complex models. Afterward, I will keep you updated! :)

Definitely @ktoprakucar. Someone mentioned this library in our discord server: https://github.com/outlines-dev/outlines
asking "Has anyone looked at this project (or other JSON constraining project) to help non-openai models output json?"

Not sure if this will help. Here is our discord, come join us :) https://discord.com/invite/a3K9c8GRGt
Repository: confident-ai/deepeval
Issue: SummarizationMetric and empty set of assessment_questions
Root cause: Documentation Deficiency

Hi, 
I am using the **SummarizationMetric** to evaluate the output of a LLM.
In the blog post example [here](https://docs.confident-ai.com/docs/metrics-summarization#example) you say that _If assessment_questions is not provided, we will generate a set of assessment_questions for you at evaluation time._

However, I wasn't able to find where these questions are generated in the code and, on my side, I always end up with `assessment_question` being `None`; thus an `inclusion_score` of 1 from this [line](https://github.com/confident-ai/deepeval/blob/51d5d2d3c337a3d3ee9a7e5005adadbc3da46a8d/deepeval/metrics/summarization.py#L163).

Could you help me on this issue?

Thanks

Edit: I am using latest version `0.20.68`

Hey @qgrail ! Yes, we got complaints that the generated questions were too arbitrary and so removed it for the time being. It is the blog that needs updating (or unless we have a better way to make the generated assessment questions more aligned with expectations)  

Understand, thanks for the clarification.

@qgrail actually bringing the inclusion score back tomorrow with auto generated assessment questions, but also renaming it to coverage score (inclusion will be used for diversity & inclusion instead)
Repository: confident-ai/deepeval
Issue: Trainer needs some further definition in DeepEvalHuggingFaceCallback
Root cause: Validation Gap

**Describe the bug**
The definition of `trainer` in `DeepEvalHuggingFaceCallback` implies optionality,
`            trainer: Trainer = None,`
but, the same `__init__` call checks for trainer args further down:
`            total_train_epochs = self.trainer.args.num_train_epochs`



**To Reproduce**
Steps to reproduce the behavior:
1. Simply instantiate DeepEvalHuggingFaceCallback without a trainer assigned in the kwargs.

**Expected behavior**
I don't know enough to know what the expected behaviour is: The code is implicitly saying that a trainer is both optional and not. What is the design aim here? Should users just be forced to provide a Trainer else raise an exception? Does this exist in other callbacks?

**Screenshots**

![Screenshot from 2024-02-21 15-57-37](https://github.com/confident-ai/deepeval/assets/155632595/7e764cd1-97b9-4514-b540-99e65f62529b)
![Screenshot from 2024-02-21 16-03-08](https://github.com/confident-ai/deepeval/assets/155632595/9706fa24-4578-402a-b472-d4befacdc1f2)



Hey @judebowyer, i think its just a small mistake, the fix would be to remove the `= None` at the end. Does it work if you provide a trainer?

@penguine-ip Yes, it does. Definitely just a tiny issue!

@judebowyer Regardless I fixed it, in the newest release :)

Come join discord too: https://discord.com/invite/a3K9c8GRGt
Repository: confident-ai/deepeval
Issue: 2KAttributeError: 'AIMessage' object has no attribute 'find'
Root cause: Interface Contract Mismatch

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**

```
import openai
from deepeval.models.base import DeepEvalBaseLLM
from azure.identity import AzureCliCredential
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from langchain.chat_models import AzureChatOpenAI
from deepeval.metrics import AnswerRelevancyMetric, ContextualRelevancyMetric
from deepeval.metrics import HallucinationMetric


class DeepEvalAzureOpenAI(DeepEvalBaseLLM):
    def __init__(
        self,
        model
    ):
        self.model = model

    def load_model(self):
        return self.model

    def _call(self, prompt: str) -> str:
        chat_model = self.load_model()
        return chat_model.invoke(prompt)

    def get_model_name(self):
        return "Custom Azure OpenAI Model"


TENANT_ID = "xyzzy"
OPEN_AI_SCOPE = 'https://cognitiveservices.azure.com/.default'
MODEL_NAME = 'gpt-4-32k'
openai.api_base = 'https://xyz.azure.com/' 
openai.api_version = '2023-05-15'
 
# for AD:
openai.api_type = 'azure_ad'
token_credential = AzureCliCredential(tenant_id=TENANT_ID)
token = token_credential.get_token(OPEN_AI_SCOPE)
openai.api_key = token.token
custom_model = AzureChatOpenAI(
        deployment_name='gpt-4-32k',
        model_name="gpt-4-32k",
        openai_api_key=openai.api_key,
        openai_api_base=openai.api_base,
        openai_api_version=openai.api_version
    )

azure_openai = DeepEvalAzureOpenAI(model=custom_model)


context=["A man with blond-hair, and a brown shirt drinking out of a public water fountain."]

actual_output="A blond drinking water in public."

test_case = LLMTestCase(
    input="What was the blond doing?",
    actual_output=actual_output,
    context=context
)
metric = HallucinationMetric(threshold=0.5, model=azure_openai)

metric.measure(test_case)
print(metric.score)


```
**Screenshots**
If applicable, add screenshots to help explain your problem.
<img width="1030" alt="Screenshot 2024-02-22 at 19 31 42" src="https://github.com/confident-ai/deepeval/assets/57705684/6f7b8540-bd03-4cde-b358-3af348724e0c">

**Desktop (please complete the following information):**
 - OS: macOS 14.2.1 (23C71)


**Additional context**
python package versions:
<img width="246" alt="Screenshot 2024-02-22 at 19 33 14" src="https://github.com/confident-ai/deepeval/assets/57705684/4c872157-a694-46c5-bdf9-1a5ddf2768f8">

python 3.10


duplicate of
https://github.com/confident-ai/deepeval/issues/502#issue-2149148906
Repository: confident-ai/deepeval
Issue: Issues using Custom LLM on air gapped systems.  Remove __init__.py chech_for_update()
Root cause: Validation Gap

Importing DeepEvalBaseLLM hangs on air gapped system due to chech_for_update() process during init.

commenting out the request to function at end of init allows loading of Module  and usage of custom locally loaded LLM


@JoeSF49 Thanks for bringing this up, do you know why it hangs? I'll add a timeout just in case

a timeout would be great thanks.  it's just a secured system running a
workload with all the libraries and models locally and airg apped (no
internet access)

thanks


On Thu, Feb 22, 2024 at 11:23 AM Jeffrey Ip ***@***.***>
wrote:

> @JoeSF49 <https://github.com/JoeSF49> Thanks for bringing this up, do you
> know why it hangs? I'll add a timeout just in case
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/confident-ai/deepeval/issues/506#issuecomment-1959803049>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AXIZ6GL2YBRT4TGV5QGIIO3YU5WIBAVCNFSM6AAAAABDVHGKSGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJZHAYDGMBUHE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


@JoeSF49 I see, glad to know it isn't a bug but an edge case, I'll ping you once its out tomorrow

@JoeSF49 Update is out, now has a 5 second timeout for the latest version v0.20.71

Also come join us on discord: https://discord.com/invite/a3K9c8GRGt
Repository: confident-ai/deepeval
Issue: Conversation Evaluation
Root cause: Unimplemented Feature Gap

Currently, `deepeval` only evaluates based on input output pairs. I'm hoping to get help on extending the `LLMTestCase` class so `deepeval` can start supporting evaluations of conversations.

Comment or DM on discord to discuss more: https://discord.com/invite/a3K9c8GRGt
Repository: confident-ai/deepeval
Issue: Support popular benchmarks
Root cause: Unimplemented Feature Gap

Currently, users of `deepeval` can only create their own evaluation dataset/test cases. To support more users fine-tuning their model, `deepeval` should be able to import standard benchmarks such as MMLU, hellaswag, TruthfulQA, Big Bench, etc.

Comment or DM on discord to discuss more: https://discord.com/invite/a3K9c8GRGt

+1. A paved path for generating metrics/scores covered by [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) and [LLM Safety Leaderboard](https://huggingface.co/spaces/AI-Secure/llm-trustworthy-leaderboard) (and maybe a few others) can be a great bonus for the framework

@rohinish404 We're trying to support multiple popular benchmarks for users to have instant access to through deepeval. TruthfulQA is a popular and also extremely easy to integrate: https://github.com/sylinrl/TruthfulQA/blob/main/data/v0/TruthfulQA.csv

If users can just have access to truthfulQA like this:
```python
from deepeval.benchmarks import TruthfulQA

truthful_qa = TruthfulQA()
truthful_qa.evaluate(...)
```

In .benchmarks:
```python
from deepeval.dataset import EvaluationDataset

class TruthfulQA(EvaluationDataset):
    pass
```

Does this make sense to you?

> @rohinish404 We're trying to support multiple popular benchmarks for users to have instant access to through deepeval. TruthfulQA is a popular and also extremely easy to integrate: https://github.com/sylinrl/TruthfulQA/blob/main/data/v0/TruthfulQA.csv
> 
> If users can just have access to truthfulQA like this:
> 
> ```python
> from deepeval.benchmarks import TruthfulQA
> 
> truthful_qa = TruthfulQA()
> truthful_qa.evaluate(...)
> ```
> 
> In .benchmarks:
> 
> ```python
> from deepeval.dataset import EvaluationDataset
> 
> class TruthfulQA(EvaluationDataset):
>     pass
> ```
> 
> Does this make sense to you?

Yup let me have a look at the codebase and then will ask further in discord.
Repository: confident-ai/deepeval
Issue: [BUG]: v0.20.69

Looks like a bug was introduced in v0.20.69

```
  File "/opt/app-root/lib64/python3.11/site-packages/deepeval/__init__.py", line 9, in <module>
    from deepeval.evaluate import evaluate, run_test, assert_test
  File "/opt/app-root/lib64/python3.11/site-packages/deepeval/evaluate.py", line 11, in <module>
    from deepeval.metrics import BaseMetric
  File "/opt/app-root/lib64/python3.11/site-packages/deepeval/metrics/__init__.py", line 4, in <module>
    from .bias.bias import BiasMetric
ModuleNotFoundError: No module named 'deepeval.metrics.bias
```

@AndresPrez Fixed! thanks for catching it, latest version 0.20.70

Come join our discord, someone also asked aboiut this: https://discord.com/invite/a3K9c8GRGt
Repository: confident-ai/deepeval
Issue: Missing dependency: detoxify
Root cause: Environment Incompatibility

**Describe the bug**
ModuleNotFoundError: No module named 'detoxify'

**To Reproduce**
Run a python script with DeepEval imports

**Expected behavior**
All needed modules should be included in dependencies

**Screenshots**
Traceback (most recent call last):
  File "---", line 33, in <module>
    from deepeval.models.base import DeepEvalBaseLLM
  File "---/lib/python3.10/site-packages/deepeval/models/__init__.py", line 8, in <module>
    from deepeval.models.detoxify_model import DetoxifyModel
  File "---/lib/python3.10/site-packages/deepeval/models/detoxify_model.py", line 3, in <module>
    from detoxify import Detoxify
ModuleNotFoundError: No module named 'detoxify'

**Desktop (please complete the following information):**
 - OS: Linux

**Smartphone (please complete the following information):**




@moruga123 have you tried `pip install detoxify`?

also are you trying to use the detoxify metric? the latest version does not require detoxify


This is working after changing to a clean venv with short requirement list

@moruga123 Glad it worked out!
Repository: confident-ai/deepeval
Issue: Allow users to configure embeddings for RAGAS metrics 
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
The versions of `deepeval` and `ragas` that I'm using:
- deepeval==0.20.70
- ragas==0.1.1

I was trying to use `RAGASAnswerRelevancyMetric` to evaluate the generated output of my Microsoft Azure Endpoints for the Open AI GPT models. Under the hood, in `ragas/evaluation.py`, embeddings are used for the calculation, and I cannot find a way to override the embeddings with the deepeval library. ragas library supports Microsoft Azure endpoints - their instruction:
"In order to use the library with Microsoft Azure endpoints, you need to set the OPENAI_API_TYPE, OPENAI_API_BASE, OPENAI_API_KEY and OPENAI_API_VERSION. The OPENAI_API_TYPE must be set to 'azure' and the others correspond to the properties of your endpoint. In addition, the deployment name must be passed as the model parameter."
I'm not sure how I can pass the deployment name as the model parameter to ragas from deepeval.

**Describe the solution you'd like**
I hope users of deepeval can have a better control over how embeddings are generated when using ragas-related metrics.

**Describe alternatives you've considered**
Not applicable

**Additional context**
Not applicable

P.S. thanks for open-sourcing this great library.


@Li0425 Hey! No problem, we'll support it

@Li0425 It's done! Newest version 0.20.71, here are the docs: https://docs.confident-ai.com/docs/metrics-ragas#example

Note that only the RAGASAnswerRelevancy metric uses embeddings, it doesn't matter to the implementation but just as a piece of info :)

Come join our discord to talk :) : https://discord.com/invite/a3K9c8GRGt

Hi @penguine-ip, thanks for the super quick update! Joined the discord 👍 

@Li0425 no problem :) and happy to have u

Hello @penguine-ip , regarding this change: for answer relevancy it works well. However, for other ragas metrics that don't really calculate based on embeddings, embedding_factory() in ragas library would be hit anyways, and deepeval test couldn't be run again due to not able to configure combeddings

@Li0425 Can you give an example? I thought only Answer relevancy uses embeddings. I tried using the RAGAS metric and didn't run into errors. 

Also come to discord if you're on there! https://discord.com/invite/a3K9c8GRGt I check it a lot more often
Repository: confident-ai/deepeval
Issue: Evaluation Dataset Generation
Root cause: Unimplemented Feature Gap

Evaluation dataset generation is coming to deepeval by end of this week. For this feature, we're looking at the following:

1. Allow users to generate test cases based on their knowledge base
2. Allow users to choose how to chunk their knowledge base 
3. Allow users to specify how many test cases to generate 
4. Allow users to complicate test cases to make them more realistic (https://arxiv.org/pdf/2304.12244.pdf, https://mlabonne.github.io/blog/notes/Large%20Language%20Models/phi1.htm) 

Feedback, suggestions, and contributions for any of the points above are welcomed 😊

link not found:
https://mlabonne.github.io/blog/notes/Large%20Language%20Models/phi1.htm

@penguine-ip Is this still needed ? new to this repo trying to get some good first issues
Repository: confident-ai/deepeval
Issue: Make multithreading optional
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
There's some multithreading logic inside some of deepeval's metrics. Some accept the multithreading flag to make that optional.

**Describe the solution you'd like**
Similar to what's already implemented for metrics `HallucinationMetric` and `FaithfulnessMetric`, it would be nice to also have the ability to turn multithreading off for the rest of the metrics such as `ContextualRelevancyMetric`.

**Additional context**
The ask is simply because Im computing different metrics on different test cases that Im sending to multiple threads, but metrics such as `ContextualRelevancyMetric` is creating multiple threads itself from an already secondary thread.

Also done, thank you so much :) @AndresPrez I didn't forget about making all metrics for a single test case run async, just taking a bit longer since theres more potential errors to consider

^taking longer than expected, hopgin to have it out tomorrow

@AndresPrez just added this feature (v0.20.85). Switched over from threading to async execution. If you're using a custom model, you'll have to make tiny modifications to implement an async method to generate model outputs: https://docs.confident-ai.com/docs/metrics-introduction#using-a-custom-llm

Apart from this, you should see a lot of speed ups without having to make any changes.

@AndresPrez anticipating some bugs though, so please let me know if you run into any!

@penguine-ip I've looked at the code changes, looks good! I haven't had time to test it out yet but I will soon!

@AndresPrez thanks! Let me know if you run into any errors, one thing that some users are encountering is now rate limits since they run too concurrently...

> @AndresPrez thanks! Let me know if you run into any errors, one thing that some users are encountering is now rate limits since they run too concurrently...

Facing this issue with synthesizer when giving it an input of many documents to embed. The solution I did was synthesize subset by subset but it would be good for this to be handled internally.

@EssamWisam are you using your own model? Also if you come to our discord we will reply a lot faster.
Repository: confident-ai/deepeval
Issue: Type Hinting issue with python<=3.8
Root cause: Environment Incompatibility

**Describe the bug**
Built-in collection types such as list and dict don't work with python<=3.8 for Type Hinting Generics. 

**To Reproduce**
Steps to reproduce the behavior:
1. Create a virtual environment with python <= 3.8
2. Try to log in using `deepeval login`
3. You'll see an error: `TypeError: 'type' object is not subscriptable`

**Expected behavior**
Typing should work with python<=3.8.

**Error Traceback**
```
Traceback (most recent call last):
   File "/venv/bin/deepeval", line 5, in <module>
     from deepeval.cli.main import app
   File "/venv/lib/python3.8/site-packages/deepeval/\_\_init__.py", line 8, in <module>
     from deepeval.evaluate import evaluate, assert_test
   File "/venv/lib/python3.8/site-packages/deepeval/evaluate.py", line 15, in <module>
     from deepeval.test_run import test_run_manager, APITestCase, MetricsMetadata
   File "/venv/lib/python3.8/site-packages/deepeval/test_run/\_\_init__.py", line 1, in <module>
     from .test_run import (
   File "/venv/lib/python3.8/site-packages/deepeval/test_run/test_run.py", line 65, in <module>
     class TestRun(BaseModel):
   File "/venv/lib/python3.8/site-packages/deepeval/test_run/test_run.py", line 81, in TestRun
     hyperparameters: Optional[dict[Any, Any]] = Field(None)
TypeError: 'type' object is not subscriptable
```

@navkar98 thanks so much, looking forward to it!

@penguine-ip I created a PR but it failed as I didn't had api key to pass. Can you let me know what can I do about it?

@navkar98 no problem, everything else is working so i just merged it, thanks a lot! New release out in 1-2 hours.
Repository: confident-ai/deepeval
Issue: Store computed metrics in a dataframe
Root cause: Unimplemented Feature Gap

Objective:
I have many test cases - query, response, context trios in a pandas dataframe. I create an llm test case per trio.. however the outcome of bulk testing locally is a json file.

Outcome:
save the responses in the same dataframe while computing the scores. this should be asynchronously done. and the json file should be used only for validation purpose.

Hey @piseabhijeet , can you show an example? I get what you need, but i'm not sure how would it write to the same dataframe.
Repository: confident-ai/deepeval
Issue: Pydantic v1 fallback for this part of the code
Root cause: External Dependency Breakage

I've seen other part of the code handle pydantic v1 case gracefully but this fails

https://github.com/confident-ai/deepeval/blob/97f528a9336b2cde9a2ba30787a25ac4331ddb37/deepeval/test_run/test_run.py#L95

hey @kubre thanks for bringing this up! Do you know how it would look like? Happy to quickly approve a PR if you make one otherwise it will probably be next week before i can get around to this.

https://github.com/confident-ai/deepeval/pull/600#issue-2188203483
Repository: confident-ai/deepeval
Issue: temp_test_run.json contains partial entries when running in bulk
Root cause: Resource Mishandling

When running a bulk dataset from commandline:

`deepeval test run src.py -n 3`

the json file that gets generated contains partial entries. and some test cases are lost even when the run was successful and summary was displayed on the console.

Is there a way to get maybe n files (per n parallel processes), or equivalent?

hey @piseabhijeet addressed on discord
Repository: confident-ai/deepeval
Issue: Improve LangChain Integration
GitHub labels: documentation

enhancement

help wanted

With the improvements in the package, LangChain guide will need to be updated to demonstrate capabilities.

- [ ] Adding a Langchain callback here

This has now been done as part of a PR into LangChain itself. 
See here https://github.com/langchain-ai/langchain/pull/10349  for initial implementation.