zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: RAGAS metric issue
Root cause: External Dependency Breakage

Hey Im trying to use the RAGAS metrics and constantly running into this issue: 
pyarrow.lib.ArrowInvalid: Could not convert <built-in function input> with type builtin_function_or_method: did not recognize Python value type when inferring an Arrow data type


Im installing these dependencies before hand:
!pip install llama-index
!pip install -U deepeval
!pip install ragas

I also got this when trying to get the overall metric: 'RagasContextualRelevancyMetric' object has no attribute 'metrics' 
With this code:
def ragasonly(query,output,expected_output,context):
  ragas_metric = RagasMetric()
  test_case = LLMTestCase(
      input=query,
      actual_output=output,
      expected_output=expected_output,
      context=context,
  )
  score = ragas_metric.measure(test_case)
  return score

Hey Lewis, can you try updating to the newest version of deepeval and show me the trace stack if the error still persists? Thanks! Also easier if you ask it on our discord: https://discord.com/invite/a3K9c8GRGt

Replied in discord, closing the issue :)
Repository: confident-ai/deepeval
Issue: Bug when following the documentation.
Root cause: Documentation Deficiency

https://docs.confident-ai.com/docs/getting-started#llm-evaluated-metrics

```
def test_summarization():
    input = "What if these shoes don't fit? I want a full refund."

    # Replace this with the actual output from your LLM application
    actual_output = "If the shoes don't fit, the customer wants a full refund."

    summarization_metric = LLMEvalMetric(
        name="Summarization",
        criteria="Summarization - determine if the actual output is an accurate and concise summarization of the input.",
        evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
        minimum_score=0.5
    )
    test_case = LLMTestCase(input=input, actual_output=actual_output)
    assert_test(test_case, [summarization_metric])
```

The documentation does not say that OPENAI_API_KEY is required for this step. The test fails with

```
 pydantic.error_wrappers.ValidationError: 1 validation error for ChatOpenAI
E   __root__
E     Did not find openai_api_key, please add an environment variable `OPENAI_API_KEY` which contains it, or pass  `openai_api_key` as a named parameter. (type=value_error)

```

Please update the docs and include the instructions.


@rommik-nuvalence you can supply the openai api key by setting it as an env variable in the terminal like this:
```
export OPENAI_API_KEY="your openai api key"
```
Repository: confident-ai/deepeval
Issue: Retrieval context removed
Root cause: Documentation Deficiency

The retrieval context is no longer part of the `LLMTestCase` as of [this commit](https://github.com/confident-ai/deepeval/commit/d94a854a3bf3464cb4f16367995784908925bd5c#diff-eaf87080789200606ea161c64f03eabd451634889dd5a33e0c506ba6ac41d616). Is this intended? I think, it makes a lot of sense to also test for the quality of the retrieved context.

If you really don't want to have that anymore, you should at least remove it from the documentation: https://docs.confident-ai.com/docs/evaluation-test-cases#retrieval-context

Hey @hguhlich, it was removed because there are currently no metrics supporting ranking similarity, and also we're no evidence so far it is being used by users. Are you using it? Happy to add it back for our release early next week.

Shouldn't the [Contextual Relevancy Metric from RAGAS](https://docs.ragas.io/en/latest/concepts/metrics/context_relevancy.html) use the retrieval context? From [your documentation](https://docs.confident-ai.com/docs/evaluation-test-cases#retrieval-context): 

> Remember, context is the ideal retrieval results for a given input and typically come from your evaluation dataset, whereas retrieval_context is your LLM application's actual retrieval results.

So, comparing the context with the input doesn't make sense if the context should be relevant by definition. We should compare the retrieved context with the input.

Definitely something I overlooked, thanks for taking the time to look through the commit history and pointing it out! The new release on Monday will contain the fixes. Also come join our discord if you're on there, would love to have you in the community: https://discord.com/invite/a3K9c8GRGt

@hguhlich https://docs.confident-ai.com/docs/evaluation-metrics#ragas, its out, thanks!
Repository: confident-ai/deepeval
Issue: Need to untangle dependencies around protobuf
Root cause: External Dependency Breakage

**Describe the bug**
A clear and concise description of what the bug is.

deepeval 0.20.24 still specify old version of protobuf so we can't install [qdrant-client](https://pypi.org/project/qdrant-client/) 1.6.9. qdrant is one of the famous vector db, which is likely installed along with LLM application. 

```
deepeval 0.20.24 requires protobuf<=3.20.5, but you have protobuf 4.25.1 which is incompatible.
```

**To Reproduce**
Steps to reproduce the behavior:

```sh
python -m venv
source ./venv/bin/activate
```

```sh
pip install deepeval==0.20.24
```

```sh
pip install qdrant-client==1.6.9
```

Then you should see the error below.

```
deepeval 0.20.24 requires protobuf<=3.20.5, but you have protobuf 4.25.1 which is incompatible.
```


**Desktop (please complete the following information):**
 - OS: OSX14.1.1 (23B81)
 - Browser no browser
 - Version: 0.20.24
 - Python: 3.11.2

**Additional context**
Add any other context about the problem here.

[This line in setup.py](https://github.com/confident-ai/deepeval/blob/main/setup.py#L28) is the root-cause.



Thanks for bringing this up @pomcho555, will version  4.23.4  work?

@penguine-ip 
Yeah, it works ignoring the dependency conflict at least on my laptop as shown in the description in this issue.

Thanks!
Repository: confident-ai/deepeval
Issue: Integration of LM Evaluation Harness. 
Root cause: Unimplemented Feature Gap

Harness being one of the general evaluation frameworks for hundreds of tasks and benchmarks on different types of metrics. 

- check LM EValuation Harness [here](https://github.com/EleutherAI/lm-evaluation-harness)

A general evaluation of LLMs on general tasks is very much important, when doing research and also doing pre-production checks. Some general tasks like toxicity, or evaluating LLMs on summarization etc, is super important. 

Since, Harness is very much CLI based, integrating this to deepeval in the form of a modular pipeline, can be super helpful for doing CI checks during fine-tuning LLMs or pre-productions steps. 


Expected things:

1. Terminal based output
2. DeepEval check 

Sounds great, please add it in this module: https://github.com/confident-ai/deepeval/tree/main/deepeval/check. The entry point should be the check function in check.py, let me know if you have any questions!

PS. you might want to put the terminal output logic inside the check function to avoid repetitive code.

> Sounds great, please add it in this module: https://github.com/confident-ai/deepeval/tree/main/deepeval/check. The entry point should be the check function in check.py, let me know if you have any questions!
> 
> PS. you might want to put the terminal output logic inside the check function to avoid repetitive code.

sounds good.
Repository: confident-ai/deepeval
Issue: Integration of HuggingFace Callback.
Root cause: Unimplemented Feature Gap

Similar to [wandb callback](https://github.com/huggingface/transformers/blob/9660e27cd0ada58b60d7e6ea0558c878c5e20580/src/transformers/integrations/integration_utils.py#L667), having a deepeval huggingface callback is super helpful in doing spontaneous fine-tuning and evaluation of LLMs on custom or pre-defined evaluations. 

So, the sample script, would look something like this:

```python
from transformers import Trainer
from transformers import TrainingArguments
from transformers.callback import deepeval callback

args = TrainingArguments(iterations=10000, callback=DeepEvalCallBack(log_every_epoch=100))
trainer = Trainer(model, args)

trainer.train()
```

Here this is a simple usage script on Fine-tuning LLMs should with deepeval.
Repository: confident-ai/deepeval
Issue: add support for Azure Openai for LLM based testing
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Currently, for LLM based testing, it uses openai. it would be better if we can evaluate based on azure openai as well.
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
add a method which accepts azure openai 
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
I could not use evalute LLM based due to this. I had to implement one myself.
**Additional context**
Add any other context or screenshots about the feature request here.


@senthilkumarimuth I can definitely do this, however I have never worked with Azure OpenAI before nor do I have an Azure OpenAI deployment.

From what I can gather and correct me if I'm wrong, I'll need `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`,  `openai_api_version`, and ` azure_deployment_name`. Is there anything I've missed?


Also please come to our discord for better communication and I can resolve the issue faster: https://discord.com/invite/a3K9c8GRGt

> @senthilkumarimuth I can definitely do this, however I have never worked with Azure OpenAI before nor do I have an Azure OpenAI deployment.
> 
> 
> 
> From what I can gather and correct me if I'm wrong, I'll need `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`,  `openai_api_version`, and ` azure_deployment_name`. Is there anything I've missed?
> 
> 

Yes that's all you need. 

@senthilkumarimuth it's out! https://docs.confident-ai.com/docs/metrics-llm-evals#using-azure-openai

Version 0.20.36

> @senthilkumarimuth it's out! https://docs.confident-ai.com/docs/metrics-llm-evals#using-azure-openai
> 
> 
> 
> Version 0.20.36

So quick. Appreciate it. Let me some test cases then. Good day!

Yes please let me know if it doesn't work, because I don't have an azure account so I couldn't test it.

@senthilkumarimuth its done, new docs are here: https://docs.confident-ai.com/docs/metrics-introduction#using-azure-openai

Please join our discord for all future questions and 10/10 customer support :) https://discord.gg/a3K9c8GRGt
Repository: confident-ai/deepeval
Issue: Upgrade `protobuf` dependency
Root cause: External Dependency Breakage

[This PR](https://github.com/confident-ai/deepeval/pull/322) upgrades `protobuf` to version 4.x but it seems that the main branch still references 3.x. Could we get `protobuf` upgraded to version 4 on main?

Hey @arturpat, it was later downgraded because one of our metrics depend on that version. But don’t worry, that metric is getting deprecated tomorrow, I will be able to upgrade it in tomorrow’s release.

Do you have discord by the way? Please join us to since we find it easier to communicate there: https://discord.gg/a3K9c8GRGt

Awesome, thank you! I do have discord but I'm not using it that actively. I'll make sure to join and check it out from time to time.

@arturpat the latest version uses x4.! Upgrade to 0.20.42 :)
Repository: confident-ai/deepeval
Issue: OpenAI key setting on GitHub or in official document 
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
When following the summarization metric example (used a default model `gpt-4-1106-preview`) from the official document [SummarizationMetric](https://docs.confident-ai.com/docs/metrics-summarization), the code returned an error of `Did not find openai_api_key, please add an environment variable 'OPENAI_API_KEY' which contains it, or pass  'openai_api_key' as a named parameter. (type=value_error)`. 

We have not find the corresponding settings from GitHub page or official document yet. 

**Describe the solution you'd like**
Can you please add some instructions about settings OpenAI Key in official examples? But please let me know if we just missed it. 

**Additional context**
Using the SummarizationMetric


Hi @Dee-Ma,

You can just add OPENAI_API_KEY into your bashrc. You can get that api key from openai website. 

@agokrani That's right! 😊
Repository: confident-ai/deepeval
Issue: Hard coded minimum score
Root cause: Architectural Constraint

when we run a test case, we want to compare the LLM response matches with expected response. if match score is greater than MINIMUMSCORE, then we consider that the test is passed. I see that minimum score is hardcoded. can this be customizable by developer?

<img width="295" alt="Capture" src="https://github.com/confident-ai/deepeval/assets/81953865/03e879a1-4590-4e3e-b7ca-7a95a1ec44c9">



Oh my goodness thanks for bringing this up, I've been trying to catch this bug forever! Will push out a release in a few hours to fix this.

okay, you just replace 0.5 with this

[image: image.png]

On Wed, Jan 3, 2024 at 4:50 PM Jeffrey Ip ***@***.***> wrote:

> Oh my goodness thanks for bringing this up, I've been trying to catch this
> bug forever! Will push out a release in a few hours to fix this.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/confident-ai/deepeval/issues/393#issuecomment-1875211740>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ATRIISOIUCCXXMZQVAJKEX3YMU5JJAVCNFSM6AAAAABBJ3KIMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVGIYTCNZUGA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


I can't see the image but yes, i think i know what you mean :) New release in an hour!

@senthilkumarimuth by the way I don't know have you seen but Azure OpenAI is not supported: https://docs.confident-ai.com/docs/metrics-introduction#using-azure-openai

Thank you.

Can you add latency as well of the metric?

On Mon, Jan 8, 2024 at 2:33 AM Jeffrey Ip ***@***.***> wrote:

> Closed #393 <https://github.com/confident-ai/deepeval/issues/393> as
> completed.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/confident-ai/deepeval/issues/393#event-11411975841>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ATRIISLF5IJJ5EJBIKLPX3DYNMESZAVCNFSM6AAAAABBJ3KIMWVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRGQYTCOJXGU4DIMI>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
Repository: confident-ai/deepeval
Issue: Rich Progress Context overwrites one another when running pytest in parallel 
Root cause: Resource Mishandling

The progress spinner (https://github.com/confident-ai/deepeval/blob/main/deepeval/progress_context.py#L23) overwrites when it is used in parallel.

Specifically at `deepeval test run <test_file>.py -n 3`

@penguine-ip Is this issue open to work?

@ptdatta hey it is resolved already!
Repository: confident-ai/deepeval
Issue: ToxicityMetric errors out when actual_output is provided
Root cause: Validation Gap

**Describe the bug**
ToxicityMetric's `measure` method seems to have an incorrect logic in the input validation. If `test_case.actual_output` exists, it errors out, but it should error out only when it's None.

i.e.
```python
        if test_case.input is None or test_case.actual_output:
            raise ValueError("Input or actual output cannot be None")
```
should be:
```python
        if test_case.input is None or test_case.actual_output is None:
            raise ValueError("Input or actual output cannot be None")
```

**To Reproduce**
Steps to reproduce the behavior:

Run the following code sample (e.g. save as `test_foo.py` and run `deepeval test run test_foo.py`):

```python
import pytest
from deepeval import assert_test
from deepeval.metrics import ToxicityMetric
from deepeval.test_case import LLMTestCase

def test_toxicity():
        metric = ToxicityMetric(threshold=0.5)

        tc = LLMTestCase(
                input="What is 2+2?",
                actual_output="4",
                retrieval_context=[],
        )
        assert_test(tc, [metric])
```

**Expected behavior**
It should not fail.

**Screenshots**
![image](https://github.com/confident-ai/deepeval/assets/119349094/32055e43-ebb5-4189-8e37-2e879ab9abbe)

**Desktop (please complete the following information):**
 - OS: MacOS (but shouldn't matter)
 - Browser: N/A
 - Version [e.g. 22]: deepeval 0.20.50

**Smartphone (please complete the following information):**
N/A

**Additional context**
A simple logic bug in code, but there doesn't seem to be any unit test associated with the code.



Thanks for bringing this up! Will change it immediately

It's fixed, newest version 0.20.51, run `pip install -U deepeval' for the update, thanks!

That was fast!
Repository: confident-ai/deepeval
Issue: Dataset cross pollution bug
Root cause: Validation Gap

**Describe the bug**
I had a hard time figuring out exactly where the bug is, but this is what I'm seeing.  

I want to create 2 datasets;

```
ds1 = EvaluationDataset()
ds2 = EvaluationDataset()
```

So far so good.  Now I make a test and add it to ds1

```
    t1 = LLMTestCase(
        id="t1",
        input="input",
        ...
    )
    ds1.add_test_case(t1)
```

Still good.  Now I add a new test to ds2

```
    t2 = LLMTestCase(
        id="t2",
        input="input",
        ...
    )
    ds2.add_test_case(t2)
```

No longer good.  Running `ds1.evaluate([metric1])` has *2* test cases in it (t1 and t2).  I expected only 1 (t1).

This is on OSX using 3.11.5 and deepeval  v0.20.51 (the previous version has the same issue)

Thank you for any help!


Aha!  I figured it out.  Because I called `EvaluationDataset()` with no argument, the default literal `[]` was used, but that default instance will be used for all new instances of EvaluationDataset.  
See https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments if I wasn't clear.
I'll make a PR, in the meantime you can do `EvaluationDataset([])` or append tests to an empty list and set `EvaluationDataset().test_cases = my_tests`.

Hey @jeffometer , I saw your PR earlier this week (merged and live) but I missed your issue. That's for the thoughtful explanation! Some contributors are also talking here: https://discord.com/invite/a3K9c8GRGt
Repository: confident-ai/deepeval
Issue: Inconsistent version between release page & pypi
Root cause: Configuration Error

**Describe the bug**
* There's an inconsistency between the [release page](https://github.com/confident-ai/deepeval/releases) version and the latest version on [pypi.com](https://pypi.org/project/deepeval/)  also https://libraries.io/pypi/deepeval shows the stale version.
* When clicked on `Homepage` option on pypi website, it throws 404, and does not link back to Confident-AI, makes the origin of the owner unknown also there's no documentation.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to https://github.com/confident-ai/deepeval/releases look for latest version
2. Compare the version on https://pypi.org/project/deepeval
3. Go to https://pypi.org/project/deepeval/ click on Homepage

**Expected behaviour**
* The package should have latest version and necessary information on the Pypi website to establish its authenticity. 

**Additional context**
Deepeval provides some really good features for Test Cases for LLMs however to use it, one needs to be sure that the latest version is up to date at all the places and is any kind of risk free also be able to establish a relationship to the Confident-AI(Package Owner)by providing necessary documentation and path to websites like homepages. Due to these inconsistencies I am unable to decide on which version to use/trust between pypi & release page.


Thanks for bringing this up, fixed!
Repository: confident-ai/deepeval
Issue: `Scorer.answer_relevancy_score` fails due to import of classes not exposed in models init
Root cause: Validation Gap

**Describe the bug**
The relevant classes are commented out in the `deepeval.models.__init__` which breaks the Scorer API for some methods. This may be due to coming changes to the API but there should probably be a warning or better documentation of `Scorer`

**To Reproduce**

```python
from deepeval.scorer import Scorer

actual_output = "My name is John."
input="What is your name?"
print(Scorer.answer_relevancy_score(predictions=actual_output, target=input))
```
```bash
UnboundLocalError: cannot access local variable 'CrossEncoderAnswerRelevancyModel' where it is not associated with a value
```


**Desktop (please complete the following information):**
 - OS: Debian 12


Currently Scorer is for internal use, and supporting it with more docs for users to plug and use will come later this month. The error is likely due to a delayed import, basically we don't want to bulk up deepeval's package size by installing more than necessary packages (since Scorer.answer_relevancy_score) is no longer used (although it still works, we found a better alternative)

Ok makes sense, thx! Feel free to close this issue