Repository: confident-ai/deepeval
Issue: error: retrieval context is None also langchain dependencies are depricated,please remove depricated dependencies it causing error
Root cause: Validation Gap

test_llm.py:None (test_llm.py)
test_llm.py:53: in <module>
    dataset.evaluate([hallucination_metric, answer_relevancy_metric])
venv\Lib\site-packages\deepeval\dataset\dataset.py:41: in evaluate
    return evaluate(self.test_cases, metrics)
venv\Lib\site-packages\deepeval\evaluate.py:124: in evaluate
    test_results = execute_test(test_cases, metrics, True)
venv\Lib\site-packages\deepeval\evaluate.py:60: in execute_test
    metric.measure(test_case)
venv\Lib\site-packages\deepeval\metrics\answer_relevancy.py:36: in measure
    raise ValueError(
E   ValueError: Input, actual output, or retrieval context cannot be None


warnings

H:\Projects_2024\deepeval\venv\Lib\site-packages\langchain\embeddings\__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.embeddings import AzureOpenAIEmbeddings`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
H:\Projects_2024\deepeval\venv\Lib\site-packages\langchain\embeddings\__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.embeddings import OpenAIEmbeddings`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
H:\Projects_2024\deepeval\venv\Lib\site-packages\langchain\chat_models\__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import AzureChatOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
H:\Projects_2024\deepeval\venv\Lib\site-packages\langchain\chat_models\__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import BedrockChat`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
H:\Projects_2024\deepeval\venv\Lib\site-packages\langchain\chat_models\__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import ChatOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
H:\Projects_2024\deepeval\venv\Lib\site-packages\langchain\chat_models\__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import ChatVertexAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
H:\Projects_2024\deepeval\venv\Lib\site-packages\langchain\llms\__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import AmazonAPIGateway`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
H:\Projects_2024\deepeval\venv\Lib\site-packages\langchain\llms\__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import AzureOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
H:\Projects_2024\deepeval\venv\Lib\site-packages\langchain\llms\__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import Bedrock`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
H:\Projects_2024\deepeval\venv\Lib\site-packages\langchain\llms\__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import OpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
H:\Projects_2024\deepeval\venv\Lib\site-packages\langchain\llms\__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import VertexAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
  
  below is my srcipt in python3.11
  
  from deepeval.test_case import LLMTestCase
import json
from deepeval import evaluate
from deepeval.metrics import HallucinationMetric, AnswerRelevancyMetric
from deepeval.dataset import EvaluationDataset

original_dataset = '''
[
  {
    "input": "how are you",
    "actual_output": "I am fine",
    "expected_output": "Yes, we offer free shipping on orders over $50.",
    "retrieval_context": "abc",
    "context": [
      "Our company operates from 10 AM to 6 PM, Monday to Friday.",
      "We are closed on weekends and public holidays.",
      "Our customer service is available 24/7."
    ]
  }
]
'''

original_dataset = json.loads(original_dataset)
print(original_dataset)

test_cases = []
for datapoint in original_dataset:
    input = datapoint.get("input", None)
    print(input)
    actual_output = datapoint.get("actual_output", None)
    print(actual_output)
    expected_output = datapoint.get("expected_output", None)
    print(expected_output)
    context = datapoint.get("context", None)
    print(context)

    test_case = LLMTestCase(
        input=input,
        actual_output=actual_output,
        expected_output=expected_output,
        context=context
    )
    print(test_case)
    print(test_case.retrieval_context)
    test_cases.append(test_case)


dataset = EvaluationDataset(test_cases=test_cases)

hallucination_metric = HallucinationMetric(threshold=0.3)
answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.5)

dataset.evaluate([hallucination_metric, answer_relevancy_metric])

# You can also call the evaluate() function directly
evaluate(dataset, [hallucination_metric, answer_relevancy_metric])


Hey, so hallucination metric is more for a fine-tuning use case whereas answer relevancy is for RAG. So for example, you're running into errors because you didn't define a `retrieval_context` parameter to your `LLMTestCase` which is required for answer relevancy, you can find the relevant part of the docs here: https://docs.confident-ai.com/docs/metrics-answer-relevancy#required-parameters

The langchain warnings are caused by the RAGAS library that we offer through our RAGAS metric, since they are not up to date. It doesn't cause any errors but just a bit annoying to look at in your terminal. I can delay the import of these metrics to get rid of the errors if you'd like.

Also join our discord for issues like this, we would love to have you to talk about evals! https://discord.com/invite/a3K9c8GRGt

Hey @p2991459 is your problem solved? 

Hi @penguine-ip 

Thanks for the clarification, Yes I am able to test it with RAG.

You're welcome!