zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: Message about nest_asyncio Should Not Be Printed
Root cause: Validation Gap

## Context

I am attempting to use DeepEval in a Python notebook (Jupyter Notebook or Google Colab)

## Issue

Under default arguments for the metric constructor (i.e. `async_mode=True`), the following message is printed directly to standard output for every single evaluation:

```
Event loop is already running. Applying nest_asyncio patch to allow async execution...
```

This is a problem because it cannot be easily suppressed or redirected, and it clutters the standard output.

## Proposed Solutions

### 1) Convert to Warning

My primary suggestion would be to modify [`utils.py`](https://github.com/confident-ai/deepeval/blob/3a5ab3e1fb9652147b981aa3691f762b67895e14/deepeval/utils.py#L47) so that instead of calling `print()`, it calls `warnings.warn()` along with an appropriate warning type. This would allow the end user to suppress this message in a more targeted way.

### 2) Add Verbosity Flag

If there is a specific reason on your end that this needs to be a `print()` statement instead of a warning, I suggest adding some form of verbosity flag to switch it on and off. Most likely that would go in the metric constructor, but I could also see it closer to where the code is actually executed (`evaluate` function or `measure` method).

## Workaround

**If anyone is getting the "Event loop is already running. Applying nest_asyncio patch to allow async execution..." message and just wants it to go away, the workaround is to turn off async execution by passing in `async_mode=False` to the metric constructor (e.g. `BiasMetric(async_mode=False)`).** Note that this will not only make the printed message go away, it will also change the way that the code runs—making the execution synchronous rather than asynchronous.


If the maintainers are on board for turning this into a warning, I can draft a PR. If you want to do something more complicated, I'm not sure if I can help with that

@hoffm386 Hey, definitely would accept a PR on this, we do weekly releases so it should be out by end of week. Thanks

This is not a bug. It is an expected behaviour. You are calling `.measure` which is a sync method from an async work flow. 

To suppresss this just use `a_measure` instead of `measure`.

Something like the below
```
async def async_eval(metric, testcase):
     output = await metric.a_measure(testcase)

     return metric

```

Then call this function like this:

```
deepeval_metric = GEval(
    name="Eval Score",
    criteria=eval_criteria,
    evaluation_params=eval_params,
    model=eval_model
)

deepeval_testcase = LLMTestCase(input=obs.input[0]['content'], actual_output=obs.output['choices'][0]['message']['content'])    

deepeval_metric = asyncio.run(async_eval(metric=deepeval_metric, testcase=deepeval_testcase))

print(deepeval_metric.evaluation_cost)
print(deepeval_metric.score)
print(deepeval_metric.reason)
```
Repository: confident-ai/deepeval
Issue: Conversation Evaluations
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I have an application with user/bot conversations. I would like to somehow apply some of the Q&A evaluations to entire conversations.

**Describe the solution you'd like**
I'd like to be able to measure conversations with LLM evaluations.

**Describe alternatives you've considered**
One idea would be to leverage an LLM to somehow condense, "distil" or extract the main user questions from the messages in the conversation and the answers to those questions given by the bot. After that, we can use the traditional Q&A metrics for each of them.
Repository: confident-ai/deepeval
Issue: Why are chunks required in FactualConsistencyMetric
Root cause: Algorithmic Error

Hey guys,

I just started today exploring you great library and was curious to understand the factual consistency metric. 

Maybe i didnt got it right, but why do we have to create chunks of our context? It seems like they have no impact at all, since 
```python 
scores = self.model.predict([(context, output), (output, context)])
```
is always called with context and output, hence, producing the same scoring results in each loop. The `max_score` can be found in the first loop iteration allready. 

Code: deepeval/metrics/factual_consistency.py:19-32
```python
def measure(self, output: str, context: str):
    context_list = chunk_text(context)
    max_score = 0
    for c in context_list:
        scores = self.model.predict([(context, output), (output, context)])
        print(scores)
        # https://huggingface.co/cross-encoder/nli-deberta-base
        # label_mapping = ["contradiction", "entailment", "neutral"]
        softmax_scores = softmax(scores)
        score = softmax_scores[0][1]
        if score > max_score:
            max_score = score

        second_score = softmax_scores[1][1]
        if second_score > max_score:
            max_score = second_score
```

Good question. 
With NLI models, we chunk it because the score of the NLI models decrease as the length of the text decreases. 
This is a known phenomenon that can be seen in this paper. https://arxiv.org/pdf/2111.09525.pdf 

The loop in the `context_list` , however is a bug. The `context` needs to be `c`. Thanks for reporting that - will push a fix soon. 

Note to self: Also make sure to reference that paper both in the documentation and implementation. 

We've added the reference paper to the documentation and fixed it - released with version 0.12.4 :)
Repository: confident-ai/deepeval
Issue: AuthenticationError
Root cause: Documentation Deficiency

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt
Yes

**Describe the bug**
 I get the error info:
AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-*** You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}
I use the base_url setting, but where can  I set the base_url? I did not find the method in toturial.

**To Reproduce**
Steps to reproduce the behavior:
1. My script is:
os.environ["OPENAI_API_KEY"] = "xxxxx"
from deepeval.metrics import SummarizationMetric
metric = SummarizationMetric(threshold=0.5)
metric.measure(test_case)
print(metric.score)
print(metric.reason) 
print(metric.is_successful())
2. Run the script
3. See error

**Expected behavior**
How can I set the OPENAI configure "base_url" in the deepeval SDK?

**Screenshots**
If applicable, add screenshots to help explain your problem.
<img width="1008" alt="image" src="https://github.com/confident-ai/deepeval/assets/3983079/009d5975-ba1a-457f-9324-521d9007b196">


@AprilJoy Try calling the openai API on its own with this api key and see if you get the same error

@penguine-ip 

I used this and can get the correct respond.

import os
import json
from openai import OpenAI
messages= [ {"role": "user", "content": "你好，一个高级工程师想学习 Azure，他已经有 10 年的开发经验，可以用哪些资料做参考"} ]

client = OpenAI(
    # api_key=os.environ.get("OPENAI_API_KEY"),
   api_key= api_key,
   base_url = "https://burn.hair/v1" 
)

response = client.chat.completions.create(model="gpt-3.5-turbo",
                                        messages=messages
                                        )
print(response.choices[0].message)

I encounter this problem too. I can't find a place to set base_url for api_key.
Repository: confident-ai/deepeval
Issue: Update to tenacity 8.4.1
Root cause: External Dependency Breakage

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

The following bug has been fixed with tenacity 8.4.1

https://github.com/langchain-ai/langchain/issues/22972


@FrancoisMasson1990 do you mean update the pip dependencies required by deepeval (specifically tenacity) to 8.4.1 because of the same bug you're encourting on the langchain thread?

I would like a tenacity update because it's currently impossible to have the most recent `weave` (from wandb) and deepeval in the same repo:

```
ERROR: Unable to find a resolution for tenacity
because of the following conflicts:
  tenacity!=8.4.0,>=8.3.0 (from weave@0.50.7)
  tenacity<9.0.0,>=8.2.3 (from instructor@1.3.0)
  tenacity~=8.2.3 (from deepeval@0.21.65)
```

Yes, basically 

> @FrancoisMasson1990 do you mean update the pip dependencies required by deepeval (specifically tenacity) to 8.4.1 because of the same bug you're encourting on the langchain thread?

Yes exactly. There is mismatch version conflict that can be solved by relaxing and updating the tenacity version.
Can you do the same with protobuf and upgrade to 5.x version ?

Hi @penguine-ip can you again the constraints on some of the library dependancies in the `setup.py` specially grpc-io


```
"docx2txt~=0.8",
        "importlib-metadata>=6.0.2",
        "tenacity~=8.4.1",
        "opentelemetry-api~=1.24.0",
        "opentelemetry-sdk~=1.24.0",
        "opentelemetry-exporter-otlp-proto-grpc~=1.24.0",
        "grpcio~=1.63.0", 
```
Repository: confident-ai/deepeval
Issue: Bug in G-Eval metrics when calculating the weighted_summation_score
Root cause: Algorithmic Error

**Describe the bug**
https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/g_eval/g_eval.py#L370C17-L370C54
<img width="723" alt="image" src="https://github.com/confident-ai/deepeval/assets/53935948/44f3cec7-51b0-414b-9e6c-97b159a09971">

In the No.367 line, when there is duplicate score in the `score_logprobs`, the previous score will be replaced while the `sum_linear_probability` still add new prob. 

For example, ('8', '0.01') and ('8', '0.001') are both in the score_logprobs,
after traversing ('8', '0.01'), token_linear_probability['8'] = 0.01, and sum_linear_probability=0.01
when coming to ('8', '0.001'), token_linear_probability['8'] = 0.001, and sum_linear_probability=0.011.
This will leads to the normalization error in the No.378 line.
Repository: confident-ai/deepeval
Issue: Disable Update Warnings
Root cause: Unimplemented Feature Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Is your feature request related to a problem? Please describe.**
I am always frustrated with the warnings to update deepevals like every day or two days.
**Describe the solution you'd like**
Just like you give us the option to opt out of telemetry, give us the option to opt out of the update warnings 
Thanks

Hey @murnitur we're pretty occupied right now but happy to merge any PRs that follow the same pattern as the telemetry opt out you described :)


Set environment variable DEEPEVAL_UPDATE_WARNING_OPT_OUT to YES to silence update warnings
```console
export DEEPEVAL_UPDATE_WARNING_OPT_OUT="YES"
```
Repository: confident-ai/deepeval
Issue: Error when generating an evaluation dataset according to the official documentation
Root cause: Validation Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
[Unsupported data type] error occurs when generating an evaluation dataset by providing document_paths according to the official documentation

**To Reproduce**
When I write code according to the official documentation（the link is https://docs.confident-ai.com/docs/evaluation-datasets#generate -an-evaluation-dataset）
```
from deepeval.dataset import EvaluationDataset
dataset = EvaluationDataset()

dataset.generate_goldens_from_docs(
    document_paths=['sample.txt'],
    max_goldens_per_document=2
)
dataset.save_as("csv","D:\project\FlybotEffectTest\eval")
```


4. See error
When executing the above code, an error occurred: 
sample.txt generated an exception: Unsupported data type
**Screenshots**
![image](https://github.com/confident-ai/deepeval/assets/139734313/443cae72-16a4-426a-ad63-b9f0a1469e0b)

![image](https://github.com/confident-ai/deepeval/assets/139734313/8f45aadc-a89f-47c5-90d1-7f8af1b4b688)


**Desktop (please complete the following information):**
 - OS: Windows10
Repository: confident-ai/deepeval
Issue: Parallelized Tests
Root cause: Unimplemented Feature Gap

Support running parallelized tests with `pytest xdist`. 

Desired developer experience: 

```bash
# Distribute the number of workers to 4
deepeval test run test_sample.py -num_workers 4
```

**Motivation:**
One problem we're running into is it's currently taking too long to run unit tests for LLMs, and it's bad developer experience for engineers to wait for 5-10 minutes to get their test results.

For example, it takes around 30 seconds to generate a LLM output depending on the length of the response, and simply running 10 test (which isn't a lot) takes quite a while. 

We're hoping to fix this with this issue.
Repository: confident-ai/deepeval
Issue: Enhancing DeepEval's Vertex AI implementation sample
Root cause: Documentation Deficiency

DeepEval docs have a Google Vertex AI implementation sample here:
https://docs.confident-ai.com/docs/metrics-introduction#google-vertexai-example

However, it has a number of issues:

1. It doesn't seem to subclass the base class correctly
2. It unnecessarily relies on LangChain to call Vertex AI
3. It unnecessarily exposes the underlying library and its safety settings outside the `GoogleVertexAI` class

I implemented another version here that fixes these issues:
https://github.com/meteatamel/genai-beyond-basics/tree/main/samples/evaluation/deepeval/vertex_ai

I have the two variants: one that uses Vertex AI directly, one that uses Vertex AI via LangChain.

I also have 3 samples how to use it here:
https://github.com/meteatamel/genai-beyond-basics/tree/main/samples/evaluation/deepeval

I'm hoping the DeepEval documentation can be updated to use these implementations instead. 

@meteatamel Thanks for the comments, will definitely update it :)
Repository: confident-ai/deepeval
Issue: "deepeval login --api-key" throws error “TypeError: Option() missing 1 required positional argument: 'default’"
Root cause: Interface Contract Mismatch

**Describe the bug**
`deepeval login --api-key VALIDKEY` fails with error:

```
Traceback (most recent call last):
  File "/opt/conda/bin/deepeval", line 5, in <module>
    from deepeval.cli.main import app
  File "/opt/conda/lib/python3.10/site-packages/deepeval/cli/main.py", line 19, in <module>
    typer.Option(
TypeError: Option() missing 1 required positional argument: 'default'
```

**To Reproduce**
Create and launch a docker image on a host with an Ampere GPU (A10g, ...) and `--platform linux/amd64`

Dockerfile
==
```
FROM ghcr.io/predibase/lorax:latest 
SHELL ["conda", "run", "-n", "base", "/bin/bash", "-c"]
WORKDIR /app
COPY requirements.txt .
RUN pip3 install --no-cache-dir --upgrade pip && pip3 install --no-cache-dir -r requirements.txt
COPY entrypoint.sh /deepeval-entrypoint.sh
ENV PYTHONUNBUFFERED 1
ENTRYPOINT [ "/deepeval-entrypoint.sh" ]
```

entrypoint.sh
==
```
#!/bin/bash
set -eEuo pipefail
deepeval login --api-key ${DEEPEVAL_KEY}
```

requirements.txt
==
```
deepeval==0.21.65
```

**Expected behavior**
Login success.

**Host:**
> uname -a
Linux b0c7edc50b19 5.15.0-1049-aws #54~20.04.1-Ubuntu SMP Fri Oct 6 22:04:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

> python -V
Python 3.10.14

> env
```
NVIDIA_VISIBLE_DEVICES=all
PYTHONUNBUFFERED=1
HOSTNAME=309a231f9f0a
NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
OPENAI_API_KEY=xxx
HUGGING_FACE_HUB_TOKEN=yyy
PWD=/app
CONDA_PREFIX=/opt/conda
PORT=80
NVIDIA_DRIVER_CAPABILITIES=compute,utility
DEEPEVAL_KEY=jjj
NV_CUDA_CUDART_VERSION=12.1.55-1
HOME=/root
CUDA_VERSION=12.1.0
HF_HUB_ENABLE_HF_TRANSFER=1
MODEL_ID=mistralai/Mistral-7B-Instruct-v0.3
HUGGINGFACE_HUB_CACHE=/data
SHLVL=1
NVARCH=x86_64
NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
PATH=/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env
```

**Additional context**
This is the only platform that i have been able to reproduce this, it works fine locally on my Mac.
By the way, i have a patch for this which i will submit.
Repository: confident-ai/deepeval
Issue: The Colab notebook left as an example in the LLamaIndex Docs is not working! 
Root cause: Documentation Deficiency

ModuleNotFoundError                       
Traceback (most recent call last)
[<ipython-input-4-a192f71549b0>](https://gmi4nx97isl-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240711-060152_RC00_651359636#) in <cell line: 1>()
----> 1 from deepeval.integrations.llamaindex import DeepEvalFaithfulnessEvaluator
      2 
      3 # An example input to your RAG application
      4 user_input = "What is LlamaIndex?"
      5 

ModuleNotFoundError: No module named 'deepeval.integrations.llamaindex'

@reemabdelrazek30 Hey! Is this the latest deepeval version (v0.21.65) ?

deepeval.integrations.llamaindex --> deepeval.integrations.llama_index

**### ModuleNotFoundError: No module named 'deepeval.integrations.llamaindex'**
Is there any update?

```
# ! pip install -q -q llama-index
# ! pip install -U -q deepeval

from deepeval.integrations.llamaindex import DeepEvalFaithfulnessEvaluator

# An example input to your RAG application
question = "............"

# LlamaIndex returns a response object that contains
# both the output string and retrieved nodes
response = documents_query_engine.query(question)

evaluator = DeepEvalFaithfulnessEvaluator()
evaluation_result = evaluator.evaluate_response(
    query=question, response=response
)
print(evaluation_result)


---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[29], line 1
----> 1 from deepeval.integrations.llamaindex import DeepEvalFaithfulnessEvaluator
      3 # An example input to your RAG application
      4 question = "Can you explain how the legislative powers are distributed in the United States and what bodies make up this structure?"

ModuleNotFoundError: No module named 'deepeval.integrations.llamaindex'
```
Repository: confident-ai/deepeval
Issue: Stuck while suing summarisation metric
Root cause: Environment Incompatibility

I was testing my Hindi summarization model and while calculating the evaluation metric for SUMMARIZATION I ran the following cell, but it kept on running for too long and did not give me any output. I have used GPT-3.5-turbo as my evaluation model. So the question is - does Deepeval not support evaluation for Hindi summarization models or I am doing something wrong? 
![Screenshot 2024-07-19 034348](https://github.com/user-attachments/assets/9aa73c78-ed18-4a71-850c-4aa80101a8da)


@Vedant336Neekhra your openai is probably rate limited, that's why you're stuck
Repository: confident-ai/deepeval
Issue: TypeError: Synthesizer.generate_goldens() got an unexpected keyword argument 'enable_breadth_evolve'
Root cause: Interface Contract Mismatch

Error trying to run the dataset creation with the error: 
TypeError: Synthesizer.generate_goldens() got an unexpected keyword argument 'enable_breadth_evolve'


**To Reproduce**
When trying to do the tutorial example code : https://docs.confident-ai.com/docs/evaluation-datasets-synthetic-data
```python
dataset.generate_red_teaming_goldens(
    synthesizer=synthesizer,
    contexts=[
        ["The Earth revolves around the Sun.", "Planets are celestial bodies."],
        ["Water freezes at 0 degrees Celsius.", "The chemical formula for water is H2O."],
    ]
)

``` 
All of the dataset.generate functions seem to have the same error

Yeah, i am getting the same stuff. 

@harvey1992 @MickeySmillieEmpirisys the docs are updated to reflect this: https://docs.confident-ai.com/docs/evaluation-datasets-synthetic-data#2-generating-from-provided-contexts
Repository: confident-ai/deepeval
Issue: Typo in bias metric (too many values to unpack (expected 2))
Root cause: Validation Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
I think there is a typo in the bias metric that is causing some issues, here: https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/bias/bias.py#L164

I think it should be just `res = self.model.generate(prompt)` since we are not using the native model

**To Reproduce**
Steps to reproduce the behavior:
1. Create a custom client from `DeepEvalBaseLLM`
2. Execute the bias metric
3. See error ( `too many values to unpack (expected 2)` )

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Smartphone (please complete the following information):**
 - Device: [e.g. iPhone6]
 - OS: [e.g. iOS8.1]
 - Browser [e.g. stock browser, safari]
 - Version [e.g. 22]

**Additional context**
Thank you in advance, you are doing a great job!!!!


Hey @damagode , thanks for the error reporting, i'm wondering do you have the error message in full?

@penguine-ip 
```
{ValueError}ValueError('too many values to unpack (expected 2)') 
```

@damagode i see thanks for bringign it up, fixing !