zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: add_test_cases_from_json_file method creates retrieval_context from context_key_name parameter.
Root cause: Algorithmic Error

**Describe the bug**
The `EvaluationDataset` method `add_test_cases_from_json_file` uses `context_key_name` parameter to create the `retrieval_context` for every `test_case`. `retrieval_context_key_name` parameter is then useless.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to deepeval/dataset
2. Click on dataset.py
3. Scroll down to line 228
4. See error `retrieval_context = json_obj.get(**context_key_name)**`

**Expected behavior**
This method should create `retrieval_context` from `retrieval_context_key_name` parameter.
 `retrieval_context = json_obj.get(retrieval_context_key_name)`

Thanks!



fixed in latest release
Repository: confident-ai/deepeval
Issue: Error handling with customLLM models in metrics classes, returning a default option instead of throwing an error
Root cause: Validation Gap

**Is your feature request related to a problem? Please describe.**
When using a custom LLM, the metrics classes that use json outputs may throw an error due to incorrect json formatting. 

**Describe the solution you'd like**
These errors could be handled by repairing the json string into the correct formatting.

**Describe alternatives you've considered**
Other alternatives may include returning a default score or empty score for cases when error triggers. This could be documented instead of assuring that custom LLMs may not work.

**Additional context**
This may make the application more ready to use for production environments with no default LLM model.


Hey @AlexisGitHu that's for the suggestion. May I understand what problems you're running into/what are you trying to achieve by having the metrics not throw an error but instead repair the JSON string?

Also any suggestions on how to repair it?

@AlexisGitHu fixed in latest release https://docs.confident-ai.com/docs/evaluation-introduction#ignore-errors
Repository: confident-ai/deepeval
Issue: Question : Summarization metric with expected output

I want to evaluate summarization given actual summarization and run this code 

```
test_cases = [LLMTestCase(input=#text_to_summarize#, 
                          actual_output=#predicted_summary#,
                          expected_output=#actual_summary#]

dataset = EvaluationDataset(test_cases=test_cases)


metric = SummarizationMetric(
    threshold=0.5,
    model="gpt-4"
)

test_results = evaluate(dataset, [metric], print_results = False)
```

What are alignment and coverage components measure here ?

Hey @snassimr for alignment it will check for hallucination + any irrelevant fluffs in your summary, while the coverage will check for the proportion of information that is included in your summary from your original text. 

Note that coverage is generated by randomly picking information in the original text and seeing whether it is in the summary. You might find yourself more tolerating than the coverage score, so typically a summarization score of 0.8 is pretty good.

Also can you ask on discord in the future? I'm starting to lose track of the issues: https://discord.com/invite/a3K9c8GRGt
Repository: confident-ai/deepeval
Issue: Question: Using DeepEval with HuggingFace LLM
Root cause: Documentation Deficiency

Hello,

Do you have a working example of using DeepEval with HuggingFace LLM without having to download the model to the local environment (like using HuggingFaceEndpoint)?

Thanks.

@nikhilkomakula No we don't, but it should just be an API request. The important thing is the `generate` method should only accept the prompt (string) and output the generation (string)

@nikhilkomakula If you come on our discord and share some code for your HugginFaceEndpoint i can give you an example: https://discord.com/invite/a3K9c8GRGt
Repository: confident-ai/deepeval
Issue: Add browser opening flag to config
Root cause: Unimplemented Feature Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Is your feature request related to a problem? Please describe.**
I run deepeval from WSL so opening up a browser is annoying as hell each time I run something.

**Describe the solution you'd like**
Some flag that I could opt-out of opening a browser window.

**Describe alternatives you've considered**
Pushing a PR to fix this.

**Additional context**
Add any other context or screenshots about the feature request here.


@agamm hey! Feel free to make a PR to make opening browser opt out.
Repository: confident-ai/deepeval
Issue: Implement an AnswerCorrectness Metric
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
When dealing with evolving Q&A applications, it is important to have a way of evaluating if and how a golden set of question and answers changes over time. For this an Answer Correctness metric that lets an LLM compare an expected answer with an actual answer can be good solution to detect if the actual answers for the golden set question are changing or breaking.

**Describe the solution you'd like**
Add support for another deepeval metric that takes the above evaluation into account.

**Describe alternatives you've considered**
Ragas implements an AnswerCorrectness metric that uses both llms and embedding similarity. But it does not give you a reason and the fact that includes embedding similarity ruins the final score. Similarity in embeddings is not a very good idea to ensure two different answers are the same.

@AndresPrez Agreed with the embedding + similarity bit. Have you tried using GEval for this with `strict=true`? I'm sure it will give good results.

@penguine-ip g-eval is actually pretty nice! thanks for the advise ❤️ 

@AndresPrez No problem! I'm actually going to push out more tutorials soon on the documentation page, I feel like the full capabilities of deepeval mostly go undiscovered 😅 Be including Answer Correctness for sure

@penguine-ip  - thanks for the explanation on this - I too got a bit fooled just because the docs state that ragas metrics are supported, and AnswerCorrectness is one of ragas metrics - bit of internal translation to do to get there.

@gtmtech Yes but we eventually deleted most of it except for the flagship ragas metrics since they weren't very good n well maintained
Repository: confident-ai/deepeval
Issue: Bug with KnowledgeRetention
Root cause: Documentation Deficiency

https://docs.confident-ai.com/docs/metrics-knowledge-retention - when i run the code present in this link i get the following error - can someone help


```
AttributeError                            Traceback (most recent call last)
/var/folders/_c/snkdtcs90vd2rn1tkt1yqty00000gn/T/ipykernel_72497/2832353720.py in <module>
      5 metric = KnowledgeRetentionMetric(threshold=0.5,model=azure_openai)
      6 
----> 7 metric.measure(test_case)
      8 print(metric.score)
      9 print(metric.reason)

~/opt/anaconda3/lib/python3.9/site-packages/deepeval/metrics/knowledge_retention/knowledge_retention.py in measure(self, test_case)
     41 
     42     def measure(self, test_case: ConversationalTestCase):
---> 43         if len(test_case.messages) == 0:
     44             raise ValueError("Messages cannot be empty")
     45 

AttributeError: 'LLMTestCase' object has no attribute 'messages'
```

@shaleenrajkaur Check the docs again, just updated it. I'm guessing the names of 'test_case' and 'conversational_test_case' got mixed up. The error message indicates you previously defined a test case of type LLMTestCase and passed it into this metric.
Repository: confident-ai/deepeval
Issue: Add Cohere command-r model support
Root cause: Unimplemented Feature Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Is your feature request related to a problem? Please describe.**
Allow users to use Cohere api with command-r model.

**Describe the solution you'd like**
Create a CohereModel class from DeepEvalBaseLLM according to https://docs.confident-ai.com/docs/metrics-introduction#using-a-custom-llm

**Describe alternatives you've considered**
Creating a personal class and copying between projects can be struggling.

**Additional context**
Cohere offers other models and one can experiment with them.


Tests are not running because poetry does not install cohere.
Any idea how to change the poetry.lock file?

They now also released a larger, 104B parameter model: [C4AI Command R+](https://huggingface.co/CohereForAI/c4ai-command-r-plus)

> They now also released a larger, 104B parameter model: [C4AI Command R+](https://huggingface.co/CohereForAI/c4ai-command-r-plus)

I added a pull requests by inserting the class in the docs. Still waiting for reply.
This relies entirely on the cohere api, but i don't know if they allow what model to use or if the api key decides the model used.
Check out #661 PR

just merged, TYSM! Would love to have you in our discord, especially when I start merging your docs into some tutorials for our users: https://discord.com/invite/a3K9c8GRGt

> Tests are not running because poetry does not install cohere. Any idea how to change the poetry.lock file?

(dw about this, its fine)
Repository: confident-ai/deepeval
Issue: Writing of any files like cache or temp to running directory - Disable or Provide Config to set location
Root cause: Configuration Error

**Is your feature request related to a problem? Please describe.**
Using deepeval inside the docker web app and not cli so don't want it to write anything to the application directory. As of now, it tries to write temp_test_run_data.json and fails as it tries to do in the 
application root folder which you don't want the application to have any write permission over.

**Describe the solution you'd like**
Either way, to disable this behavior as I've seen there is a save_on_disk option available but not exposed through the evaluate function.
Or 
an option to set where deepeval can save cache and temp files to on the disk.

![image](https://github.com/confident-ai/deepeval/assets/20380094/bdec6032-3db3-4e27-8937-7615a6916f21)

@kubre There is now an option to not write anything. Disable write_cache: https://docs.confident-ai.com/docs/evaluation-introduction#evaluating-without-pytest
Repository: confident-ai/deepeval
Issue: Typo in faithfulness template
Root cause: Validation Gap

**Describe the bug**
Typo in metrics/faithfulness/template.py, comphrensive -> comprehensive


![Screenshot 2024-04-10 at 12 27 51 PM](https://github.com/confident-ai/deepeval/assets/28518512/f098d5b6-2bd3-44b8-8f74-1eeb4c1a95df)



@palaparthi thanks!
Repository: confident-ai/deepeval
Issue: `deepeval test run` takes default pytest `addopts` with no option to override
Root cause: Architectural Constraint

**Describe the bug**
`deepeval` inherits `pytest` `addopts` with no option to overrule them. (This means that by-default excluded evals cannot be run by `deepeval test run` under any circumstance.)

**To Reproduce**

1. Mark an arbitrary test `@pytest.mark.llm`
2. Add to `pyproject.toml` the following section:
```toml
[tool.pytest.ini_options]
markers = [
    "llm: mark tests with LLM involvement"
]
addopts = "-m 'not llm'"

```
4. Run `pytest` to check that default behavior applies and test is not run
5. Run `deepeval run tests`
6. Run `pytest -m llm` to check that the behavior is inverted
7. Run `deepeval test run -m llm`, or `deepeval test run path/to/excluded/test` to attempt (& fail) to invert the behavior with `deepeval`

**Expected behavior**
I'd expect that `deepeval` will allow me to overrule pytest default behavior with `-m` and other similar pytest-like switches (e.g. `-k`).

**Desktop (please complete the following information):**
 - OS: macOS 14.4.1

**Additional context**
- Python: 3.11.8
- pytest: 8.1.1
- deepeval: 0.21.26

@shippy Thank you so much!
Repository: confident-ai/deepeval
Issue: Add functionality to calculate average score
Root cause: Unimplemented Feature Gap

Add functionality to calculate average score across multiple runs. 

This ensures a fair comparison so you don't have to view the test run multiple times. 


Is this possible now?

Did you mean to compute the average of several calls, to mitigate the randomness in the Judge LLM's outputs?

AFAIK this feature isn't there. @penguine-ip is this correct?

@atreyasha Please start a new issue since this is pretty dated but we have some support for it: https://docs.confident-ai.com/docs/evaluation-introduction#repeats
Repository: confident-ai/deepeval
Issue: Add ChatGPT Synthetic Data Creation
Root cause: Unimplemented Feature Gap

Add way to create synthetic data with ChatGPT and create an `EvaluationDataset` class that allows you to process things in bulk. 

```
dataset = create_query_answer_pairs(text="""Your content goes here""",)
```

Expected output would be the evaluation dataset will be filled with `TestCase` classes. A `TestCase` should have a `query` and `expected_output`. See `deepeval/test_case.py` 

Done in this PR: https://github.com/confident-ai/deepeval/pull/74 
Released with `v0.11.0`
Repository: confident-ai/deepeval
Issue: AttributeError for variable passed to `trimAndLoadJson` when using GEval with custom model
Root cause: Interface Contract Mismatch

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Describe the bug**
A clear and concise description of what the bug is.

GEval evaluate method fails with custom model of base class `DeepEvalBaseLLM` because `res.content` is passed to `trimAndLoadJson` instead of just the variable `res`. The bug is fixed if just `res` is passed. Why is the output of custom generate expected to have the attribute `content`?

The error is:
`*** AttributeError: 'str' object has no attribute 'content'`

**To Reproduce**
Steps to reproduce the behavior:

Relevant lines:
https://github.com/confident-ai/deepeval/blob/c09a4dcbd7294cdb9541cc275ded268123f6eb7e/deepeval/metrics/g_eval/g_eval.py#L211
https://github.com/confident-ai/deepeval/blob/c09a4dcbd7294cdb9541cc275ded268123f6eb7e/deepeval/metrics/g_eval/g_eval.py#L253


**Expected behavior**
A clear and concise description of what you expected to happen.
 `trimAndLoadJson` works as expected to load the json that is returned directly from custom model's `generate`

**Desktop (please complete the following information):**
 - Debian 12



Apologies this seems resolved on your main branch so closing this issue
Repository: confident-ai/deepeval
Issue: Do you support evaluation on multi-lingual model and dataset?
Root cause: Unimplemented Feature Gap

Can I evaluate other languages other than English using this package?

@petergaoshan Depends if the evaluation model you're using supports that language. You an use any evaluation you wish using deepeval

@penguine-ip I am using a model that supports my language yet some questions generated are in English despite all text being in the other language. The solution seems to be to modify the internal prompt but this is not exposed.

@EssamWisam happy to accept a PR on this, i'm imagining an enum of languages and provide it during metrics initialization before passing it onto the prompt template. 

> @EssamWisam happy to accept a PR on this, i'm imagining an enum of languages and provide it during metrics initialization before passing it onto the prompt template.

OK I will try to find time for this. My complain was actually regarding test set generation. I'm not assuming metrics will be that language sensitive.
Repository: confident-ai/deepeval
Issue: Sub-dependency conflict with Opentelemetry
Root cause: Configuration Error

I'm trying to use poetry to install both deepeval and opentelemetry-api & opentelemetry-sdk in my virtual environment.  I would like to use both.  Currently this fails for the following reason:

`Because no versions of deepeval match >0.21.34,<0.22.0
 and deepeval (0.21.34) depends on importlib-metadata (>=7.0.2,<7.1.0), deepeval (>=0.21.34,<0.22.0) requires importlib-metadata (>=7.0.2,<7.1.0).
And because opentelemetry-api (1.24.0) depends on importlib-metadata (>=6.0,<=7.0), deepeval (>=0.21.34,<0.22.0) is incompatible with opentelemetry-api (1.24.0).
And because opentelemetry-sdk (1.24.0) depends on opentelemetry-api (1.24.0)
 and no versions of opentelemetry-sdk match >1.24.0,<2.0.0, deepeval (>=0.21.34,<0.22.0) is incompatible with opentelemetry-sdk (>=1.24.0,<2.0.0).
So, because my-project depends on both opentelemetry-sdk (^1.24.0) and deepeval (^0.21.34), version solving failed.`

Is it possible to relax deepeval's dependencies on importlib-metadata to >=6.0 <7.1.0 so that dependency resolving is possible?


@philipchung Hey let me have a look. WHat ar eyou adding opentelemetry for?

I’m using arize phoenix with llama-index for local tracing which depends on opentelemetry

@philipchung I see. We're also adding tracing to Confident AI for llamaindex specifically, out next week. In the meantime feel free to make a PR on relaxing the dependencies in setup.py and poetry 😊

merged gonna cut a release in the next 12 hours to include the change, thansk!

Thank you!  I have sensitive data that prohibits me from uploads to a web service, which is why I am using local tracing tools like Arize Phoenix, etc.

@philipchung I see, well in that case you'll like our support for any offline LLMs :) Come join our discord btw: https://discord.com/invite/a3K9c8GRGt for any other questions. Also new release is out, thanks! 

I still have a conflict in deepeval release 0.21.35 because setup.py was changed to "importlib-metadata~=6.0.2" instead of  "importlib-metadata>=6.0.2".  This limits the compatible importlib-metadata versions to >=6.0.2,<6.1.0.  Can we change it to just >=6.0.2?

Some of my other dependencies (e.g. LiteLLM) require higher version of importlib-metadata (e.g. >=6.8.0)

@philipchung Forgot why i corrected it.. but the new release should be good! v0.21.36. Please let me know if it works, I'd like to catch after dependency bugs before the week starts!

@penguine-ip Thank you!  I am now able to install the deepeval package in my environment without any dependency conflicts.
Repository: confident-ai/deepeval
Issue: [Python 3.8] Invalid syntax for Python 3.8 
Root cause: Environment Incompatibility

## Description of the bug
In deepeval\test_run\cache.py, there is an invalid type hint for Python 3.8, see : https://github.com/confident-ai/deepeval/blob/main/deepeval/test_run/cache.py#L65

Indeed, generic type hints in standard collections is a feature available starting from Python 3.9, which makes the package unsuitable for Python 3.8 projects.

## How to fix the bug
Replace `Optional[dict[str, CachedTestCase]]` with `Optional[Dict[str, CachedTestCase]]`


Just saw that the package does not officially support Python 3.8, although apart from the bug I described, it seems to work fine on Python 3.8.

@PaulWassermann  Thanks! Just cut a new release for this.  Come join our discord btw: https://discord.com/invite/a3K9c8GRGt for any other suggestions!
Repository: confident-ai/deepeval
Issue: Documentation for Chain of Thought in GEval
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
When I read the documentation about GEval, it is incomplete.

The example code shows a `criteria` and `evaluation_steps` in the same flow. Furthermore, it shows
`evaluation_steps` with a single item, which is not very logical. If you had only one step, you would
use a criteria.

**Describe the solution you'd like**
I would like to see a realistic example of how evaluation_steps is meant to be used.



@prescod I see what you mean, we didn't explain how the execution flow works for different scenarios of providing evaluation steps and criteria.. thanks for the feedback we'll include more description and examples!

@prescod fixed: https://docs.confident-ai.com/docs/metrics-llm-evals thanks!
Repository: confident-ai/deepeval
Issue: Show me the prompt!
Root cause: Documentation Deficiency

As discussed on Discord, we need to know what prompts you are serving the evaluation LLM.

https://hamel.dev/blog/posts/prompt/

I need to see the prompt to help debug when the framework fails or even for debugging my own bugs.

Once I passed `evaluation_steps = steps.split()` when I meant `evaluation_steps = steps.split("\n")`

The former turns every word into a step, and the latter turns every line into a step. My error was obvious as soon as I looked at the LLM prompt.

The "true meaning" of various built-in metrics is also easier to understand when you read the prompt.


I think it would make sense to be able to return/print the prompt for the respective metric.

Perhaps even having a getter/setter would work (the setter for when we want to make minor changes to the built in prompt without having to write a whole new metric.


@lbux : I like your idea but just to be clear, what I'm asking for is to see the literal input and output of the LLM at runtime.

I also believe that that's what the unit of caching should be.

I do also like the idea of being able to read and write the prompt, however.

Or subclass.
Repository: confident-ai/deepeval
Issue: SummarizationMetric is not cached
Root cause: Unimplemented Feature Gap

**Describe the bug**
I cannot figure out how to speed up my `evaluate`-embedding program with `use_cache=True`

**To Reproduce**
```python
$ cat tests/test_caching.py
from deepeval import evaluate
from deepeval.test_case.llm_test_case import LLMTestCase
from deepeval.metrics import SummarizationMetric


llm_response = """
Paris is the capital city of France
    """

context = [
    "Paris is the capital city of France",
]


test_case = LLMTestCase(input="Heart Rate around 79 bpm-ish", actual_output=llm_response, context=context)
results = evaluate([test_case], [SummarizationMetric()], run_async=False, use_cache=True)

$  time (python tests/test_caching.py; python tests/test_caching.py;python tests/test_caching.py; python tests/test_caching.py;)
...
1:31.98 total
```

This is about three or four times slower than running it just once.

**Expected behavior**
It should be cached somewhere and the second and later iterations should be basically free.



Use of evaluate in Monitoring system is an import use-case for us.

@prescod the summarization metric is the only metric that does not have caching

@penguine-ip  Well then I guess it's bad luck that I used that test randomly to test the caching!

I find it interesting that caching is done at the test layer and not the LLM layer. I would expect you to cache LLM inputs and outputs given that that's where all of the slow stuff happens.

And then it would automatically transfer over to all tests.

@prescod That's actually a good point, I think the reason why was because this was easier to implement. Also added the docs to let users know about summarizaiton doesn't support cache: https://docs.confident-ai.com/docs/metrics-summarization