Repository: explodinggradients/ragas
Issue: Exception raised in Job[1]: AttributeError('str' object has no attribute 'acomplete')
Root cause: Interface Contract Mismatch

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
i have tried to downgrade to ragas 0.2.7, and tried to use llama3.2 for llm and embedding but the it keep give me an error :
Exception raised in Job[51]: AttributeError('str' object has no attribute 'aembed_documents')
Exception raised in Job[25]: AttributeError('str' object has no attribute 'acomplete')


Ragas version: both 0.2.8 and 0.2.7  
Python version: 3.11.10

**Code to Reproduce**
```py
from datasets import load_from_disk
dataset = load_from_disk("data")
from ragas import EvaluationDataset

eval_dataset = EvaluationDataset.from_hf_dataset(dataset["eval"])

from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness, SemanticSimilarity
from ragas import evaluate

from ragas.llms import LlamaIndexLLMWrapper
evaluator_llm = LlamaIndexLLMWrapper("0ssamaak0/silma-v1")

from ragas.embeddings import LangchainEmbeddingsWrapper
embeddings = LangchainEmbeddingsWrapper("silma-ai/silma-embedding-matryoshka-v0.1")

metrics = [
    LLMContextRecall(llm=evaluator_llm), 
    FactualCorrectness(llm=evaluator_llm), 
    Faithfulness(llm=evaluator_llm),
    SemanticSimilarity(embeddings=embeddings)
]
results = evaluate(dataset=eval_dataset, metrics=metrics)
```

**Error trace**
with ragas 0.2.7 it didn't give me an error trace but in 0.2.8 :

--------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[6], line 7
      1 metrics = [
      2     LLMContextRecall(llm=evaluator_llm), 
      3     FactualCorrectness(llm=evaluator_llm), 
      4     Faithfulness(llm=evaluator_llm),
      5     SemanticSimilarity(embeddings=embeddings)
      6 ]
----> 7 results = evaluate(dataset=eval_dataset, metrics=metrics)

File ~\AppData\Local\anaconda31\envs\notebook_root\Lib\site-packages\ragas\_analytics.py:205, in track_was_completed.<locals>.wrapper(*args, **kwargs)
    202 @wraps(func)
    203 def wrapper(*args: P.args, **kwargs: P.kwargs) -> t.Any:
    204     track(IsCompleteEvent(event_type=func.__name__, is_completed=False))
--> 205     result = func(*args, **kwargs)
    206     track(IsCompleteEvent(event_type=func.__name__, is_completed=True))
    208     return result

File ~\AppData\Local\anaconda31\envs\notebook_root\Lib\site-packages\ragas\evaluation.py:333, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, run_config, token_usage_parser, raise_exceptions, column_map, show_progress, batch_size, _run_id, _pbar)
    329 else:
    330     # evalution run was successful
    331     # now lets process the results
    332     cost_cb = ragas_callbacks["cost_cb"] if "cost_cb" in ragas_callbacks else None
--> 333     result = EvaluationResult(
    334         scores=scores,
    335         dataset=dataset,
    336         binary_columns=binary_metrics,
    337         cost_cb=t.cast(
    338             t.Union["CostCallbackHandler", None],
    339             cost_cb,
    340         ),
    341         ragas_traces=tracer.traces,
    342         run_id=_run_id,
    343     )
    344     if not evaluation_group_cm.ended:
    345         evaluation_rm.on_chain_end({"scores": result.scores})

File <string>:10, in __init__(self, scores, dataset, binary_columns, cost_cb, traces, ragas_traces, run_id)

File ~\AppData\Local\anaconda31\envs\notebook_root\Lib\site-packages\ragas\dataset_schema.py:410, in EvaluationResult.__post_init__(self)
    408 # parse the traces
    409 run_id = str(self.run_id) if self.run_id is not None else None
--> 410 self.traces = parse_run_traces(self.ragas_traces, run_id)

File ~\AppData\Local\anaconda31\envs\notebook_root\Lib\site-packages\ragas\callbacks.py:167, in parse_run_traces(traces, parent_run_id)
    163     for i, prompt_uuid in enumerate(metric_trace.children):
    164         prompt_trace = traces[prompt_uuid]
    165         prompt_traces[f"{prompt_trace.name}"] = {
    166             "input": prompt_trace.inputs.get("data", {}),
--> 167             "output": prompt_trace.outputs.get("output", {})[0],
    168         }
    169     metric_traces[f"{metric_trace.name}"] = prompt_traces
    170 parased_traces.append(metric_traces)

KeyError: 0
**Expected behavior**

it should give me something like that:
![image](https://github.com/user-attachments/assets/68f08fe2-fc3b-4cff-af17-40c15b88a4f3)


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


**Update** 

after changing the callbacks function from:
File ~\AppData\Local\anaconda31\envs\notebook_root\Lib\site-packages\ragas\callbacks.py:167, in parse_run_traces(traces, parent_run_id)
163 for i, prompt_uuid in enumerate(metric_trace.children):
164 prompt_trace = traces[prompt_uuid]
165 prompt_traces[f"{prompt_trace.name}"] = {
166 "input": prompt_trace.inputs.get("data", {}),
--> 167 "output": prompt_trace.outputs.get("output", {})[0],
168 }
169 metric_traces[f"{metric_trace.name}"] = prompt_traces
170 parased_traces.append(metric_traces)

to this :
            for i, prompt_uuid in enumerate(metric_trace.children):
                prompt_trace = traces[prompt_uuid]
                output = prompt_trace.outputs.get("output", {})
                output = output[0] if isinstance(output, list) else output
                prompt_traces[f"{prompt_trace.name}"] = {
                    "input": prompt_trace.inputs.get("data", {}),
                    "output": output,
                }
            metric_traces[f"{metric_trace.name}"] = prompt_traces
        parased_traces.append(metric_traces)


the KeyError Traceback stopped appearing but the exception still there, and when i print the results this is what i get:
{'context_recall': nan, 'factual_correctness': nan, 'faithfulness': nan, 'semantic_similarity': nan} 


Hi, I had the same problem before, when I used `llamaindexllmwrapper`. Now it is solved, because I suddenly found that it can run easily when using `langchainLLMwrapper` in the evaluation. But make sure your llm is wrapped in langchain style.My code is as follows：
## code with issue:
```python
# ---------------------EVAL---------------------------
llm = OnlineOpenai(url=url_critic, headers=headers, model=model)
# import metrics
from ragas.metrics import (
    Faithfulness,
    AnswerRelevancy,
    ContextPrecision,
    ContextRecall,
)

# init metrics with evaluator LLM
from ragas.llms import LlamaIndexLLMWrapper

evaluator_llm = LlamaIndexLLMWrapper(llm)
metrics = [
    Faithfulness(llm=evaluator_llm),
    AnswerRelevancy(llm=evaluator_llm),
    ContextPrecision(llm=evaluator_llm),
    ContextRecall(llm=evaluator_llm),
]
# convert pandas DataFrame to Ragas Evaluation Dataset
# 确保 'reference_contexts' 列中的值是列表，而不是字符串
if df['reference_contexts'].dtype == object:
    try:
        df['reference_contexts'] = df['reference_contexts'].apply(literal_eval)
    except ValueError as e:
        print(f"Error parsing 'reference_contexts': {e}")
ragas_ds = EvaluationDataset.from_pandas(df)
print(ragas_ds) # EvaluationDataset(features=['user_input', 'reference_contexts', 'reference'], len=12)

from ragas.integrations.llama_index import evaluate

result = evaluate(
    query_engine=query_engine,
    metrics=metrics,
    dataset=ragas_ds,
    embeddings=hf_embedding,
)
# final scores
print(result)

res_df = result.to_pandas()
res_df.to_csv("llamaindex_eval_results.csv", index=False)
======================console==============================
Exception raised in Job[37]: AttributeError('OnlineOpenai' object has no attribute 'acomplete')
Evaluating:  92%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎           | 44/48 [06:26<00:22,  5.68s/it]Exception raised in Job[42]: AttributeError('OnlineOpenai' object has no attribute 'acomplete')
Evaluating:  94%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏        | 45/48 [06:26<00:12,  4.05s/it]Exception raised in Job[36]: TimeoutError()
Evaluating:  96%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏     | 46/48 [06:49<00:19,  9.59s/it]Exception raised in Job[45]: AttributeError('OnlineOpenai' object has no attribute 'acomplete')
Evaluating:  98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████   | 47/48 [06:52<00:07,  7.90s/it]Exception raised in Job[44]: AttributeError('OnlineOpenai' object has no attribute 'acomplete')
Evaluating: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 48/48 [07:29<00:00,  9.37s/it]
{'faithfulness': nan, 'answer_relevancy': nan, 'context_precision': nan, 'context_recall': nan}
```
## after change to `LangchainLLMWrapper(llm)`
```python
……
# init metrics with evaluator LLM
from ragas.llms import LangchainLLMWrapper

evaluator_llm = LangchainLLMWrapper(llm)
metrics = [
    Faithfulness(llm=evaluator_llm),
    AnswerRelevancy(llm=evaluator_llm),
    ContextPrecision(llm=evaluator_llm),
    ContextRecall(llm=evaluator_llm),
]
……
========================console===============================
(llm_testzx) (base) [root@localhost eval_RAG]# python3 RAGAS_llamaIndex_eval.py 
/home/conda/envs/llm_testzx/lib/python3.10/site-packages/pydantic/_internal/_fields.py:172: UserWarning: Field name "stream" in "OnlineOpenai" shadows an attribute in parent "LLM"
  warnings.warn(
   Unnamed: 0                                         user_input  ...                                          reference                      synthesizer_name
0           0  What is the main contribution of Touvron et al...  ...  Touvron et al. (2023a) highlight the importanc...  single_hop_specifc_query_synthesizer
1           1  What are the key architectural features of the...  ...  The Llama models are based on the decoder-only...  single_hop_specifc_query_synthesizer
2           2     Who is Tu in the research paper about Minicpm?  ...  Tu, Y. is one of the authors of the research p...  single_hop_specifc_query_synthesizer
3           3                   who is naren in xformers librar?  ...  Naren, S. is one of the contributors to the xf...  single_hop_specifc_query_synthesizer
4           4  How does the architecture of TinyLlama, an inf...  ...  TinyLlama, an inference-optimal language model...  multi_hop_abstract_query_synthesizer

[5 rows x 5 columns]
EvaluationDataset(features=['user_input', 'reference_contexts', 'reference'], len=12)
Running Query Engine:   0%|                                                                                                                                            | 0/12 [00:00<?, ?it/s]/home/conda/envs/llm_testzx/lib/python3.10/site-packages/llama_index/llms/langchain/base.py:106: LangChainDeprecationWarning: The method `BaseLLM.predict` was deprecated in langchain-core 0.1.7 and will be removed in 1.0. Use :meth:`~invoke` instead.
  output_str = self._llm.predict(prompt, **kwargs)
Running Query Engine: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [02:50<00:00, 14.19s/it]
Evaluating:  85%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                    | 41/48 [02:56<01:54, 16.29s/it]Exception raised in Job[4]: TimeoutError()
Evaluating:  92%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎           | 44/48 [03:07<00:32,  8.06s/it]Exception raised in Job[16]: TimeoutError()
Evaluating:  96%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏     | 46/48 [03:17<00:12,  6.32s/it]Exception raised in Job[20]: TimeoutError()
Evaluating: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 48/48 [03:45<00:00,  4.70s/it]
{'faithfulness': 0.8800, 'answer_relevancy': 0.9157, 'context_precision': 0.9583, 'context_recall': 0.9556}

```



Hi @waadmhj,

Were you able to resolve the issue? Thank you, @Nom1rako, for your help!


@sahusiddharth 
No not yet i faced different issues after that.
I think the problem is with ollama not sure

do keep an eye on #1170 about that
but the original issue was fixed in a recent release 

It seems the issue was answered, closing this now.

ragas  版本  0.2.14
llm_model="qwen2.5:32b"
# llm = OllamaLLM(base_url = "http://XXX", model=llm_model)
llm = ChatOllama(base_url = "http://XXX", model=llm_model,max_tokens=160000)
evaluator_llm = LangchainLLMWrapper(llm,run_config)
#测试是否调用成功
response = llm.invoke("讲一个笑话")
print(response)

embed_model="quentinz/bge-large-zh-v1.5:f32"
embeding = OllamaEmbeddings(base_url = "http://XXXX",model=embed_model)
wrapped_embeddings = LangchainEmbeddingsWrapper(embeding)

# 测试生成嵌入向量
text = "This is a test sentence."
embed_vector = embeding.embed_query(text)
以上代码测试没有问题


questions = [
    "自动化技术与应用专业的入学基本要求？"
]

correct_answers = [
    "中等职业学校毕业、普通高级中学毕业或具备同等学力"
]

samples = []

# 对每个问题进行处理
for question, correct_answer in zip(questions, correct_answers):
    res = session.ask(question, stream=True)

    # 存储 AI 给出的最终答案及检索召回的文本片段
    final_answer = ""
    retrieved_contexts = []

    for ans in res:
        final_answer = ans.content
        if ans.reference is not None:
            for ref in ans.reference:
                retrieved_contexts.append(ref['content'])
    print("final_answer:",final_answer)
    print("retrieved_contexts:",retrieved_contexts)

    # 创建 SingleTurnSample 对象
    sample = SingleTurnSample(
        user_input=question,
        retrieved_contexts=retrieved_contexts,
        response=final_answer,
        reference=correct_answer
    )
    samples.append(sample)
    
print("samples")

# 生成评估数据集
eval_dataset = EvaluationDataset(samples=samples)
print(eval_dataset)

# 选择评估指标进行评估
from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness, SemanticSimilarity
from ragas import evaluate

metrics = [
    LLMContextRecall(llm=evaluator_llm),
    FactualCorrectness(llm=evaluator_llm),
    Faithfulness(llm=evaluator_llm),
    SemanticSimilarity(embeddings=wrapped_embeddings)
]
results = evaluate(dataset=eval_dataset, metrics=metrics)
print(results)

# 设置显示所有列，打印结果
import pandas as pd

# pd.set_option('display.max_columns', None)
# print(results.to_pandas())

df = results.to_pandas()
print(df.head())
df.head().to_csv('res.csv', index=False,encoding='utf-8-sig')

![Image](https://github.com/user-attachments/assets/a514e4a3-96e7-4566-9343-f5d7350d5bc2)
评估结果也是NAN 太难受了 排查好几天了 也不知道问题