Repository: explodinggradients/ragas
Issue: AttributeError when using evaluate 
Root cause: Interface Contract Mismatch

I have checked https://docs.ragas.io/en/latest/references/evaluate/ and related issues here and couldn't resolve my bug.

# Describe the bug
When trying to use the evaluate() method from ragas, it says:
Exception raised in Job[0]: AttributeError('RetrievalQA' object has no attribute 'agenerate_prompt'),
and returns a dictionary with nans instead of the metric's values, like:
{'context_recall': nan}.

Ragas version: 0.2.6
Python version: 3.10.8

# Code to Reproduce
from datasets import load_dataset
from langchain.vectorstores import FAISS
from langchain_huggingface import HuggingFaceEmbeddings
from langchain.llms import HuggingFacePipeline
from langchain.chains import RetrievalQA
from ragas.llms import LangchainLLMWrapper
from ragas import EvaluationDataset
from ragas.metrics import LLMContextRecall
from ragas import evaluate
from langchain.docstore.document import Document


hf_emb = HuggingFaceEmbeddings(model_name="sentence-transformers/all-mpnet-base-v2",
                                                              model_kwargs={'device': 'mps'},
                                                               encode_kwargs= {'normalize_embeddings': False})

documents = ["The Eiffel Tower is in Paris, France.",
                        "The Great Wall of China is a famous landmark.",
                        "Python is a programming language widely used for data science."]

docs = [Document(page_content=text) for text in documents]

vector_store = FAISS.from_documents(docs, hf_emb)

retriever = vector_store.as_retriever(search_kwargs={'k': 2, 'score_threshold': 0.75})

llm = HuggingFacePipeline.from_model_id(model_id="gpt2", 
                                                                         task="text-generation", 
                                                                         pipeline_kwargs={"max_new_tokens": 100})

qa_chain = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)  # pipeline type

model_wrapped = LangchainLLMWrapper(qa_chain)

dataset = load_dataset("explodinggradients/amnesty_qa", "english_v3", trust_remote_code=True)

eval_dataset = EvaluationDataset.from_hf_dataset(dataset['eval'])[:3]

results = evaluate(dataset=eval_dataset, metrics=[LLMContextRecall(llm=model_wrapped)])


# Error trace
Exception raised in Job[0]: AttributeError('RetrievalQA' object has no attribute 'agenerate_prompt')
Exception raised in Job[2]: AttributeError('RetrievalQA' object has no attribute 'agenerate_prompt')
Exception raised in Job[1]: AttributeError('RetrievalQA' object has no attribute 'agenerate_prompt')


# Expected behavior
I would expect to not have these AttributeErrors, and get something like: {'context_recall': 0.7}.


# Additional context
As you can see, I used the eval_dataset using EvaluationDataset from ragas, I wrapped the llm & retriever in qa_chain and used LangchainLLMWrapper. I'm hoping I'm just missing something and it is not a bug.

Lastly, if I use model_wrapped = LangchainLLMWrapper(llm) instead of model_wrapped = LangchainLLMWrapper(qa_chain) in the LLMContextRecall(llm=model_wrapped) (metric), I get a different error, this means that something with qa_chain is wrong, but of-course I need it since it includes the RAG part, and the llm object is only llm.

Thank you very much for your help!


I figured out that the retrieval object is not needed in the evaluate, since it already done its job creating the "eval_dataset", so only an LLM is needed. Now I have a different error, which I think I will open a new issue on. Thank's