Repository: Giskard-AI/giskard-oss
Issue: RAGAS integration fails with ragas>=0.3.3 due to missing is_finished method and removed .score() API
Root cause: External Dependency Breakage

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.18.0

### OS Platform and Distribution

macOS Monterey: Version 12.7.6 (21H1320) (MacBook Pro (Retina, 13-inch, Early 2015))

### Python version

Python 3.11.14

### Installed python packages

```shell
aiohappyeyeballs==2.6.1
aiohttp==3.13.2
aiosignal==1.4.0
annotated-doc==0.0.3
annotated-types==0.7.0
anyio==4.11.0
appdirs==1.4.4
appnope==0.1.4
asgiref==3.10.0
asttokens==2.4.1
attrs==23.2.0
audioread==3.0.1
azure-ai-agents==1.2.0b6
azure-ai-inference==1.0.0b9
azure-ai-projects==1.0.0
azure-common==1.1.28
azure-core==1.36.0
azure-core-tracing-opentelemetry==1.0.0b12
azure-cosmos==4.14.1
azure-identity==1.25.1
azure-search-documents==11.6.0
azure-storage-blob==12.27.1
backcall==0.2.0
beautifulsoup4==4.12.3
bert-score==0.3.13
bleach==6.1.0
bokeh==3.4.3
bqplot==0.12.43
cachetools==6.2.1
certifi==2024.7.4
cffi==2.0.0
cfgv==3.4.0
chardet==5.2.0
charset-normalizer==3.3.2
click==8.3.0
cloudpickle==3.1.2
colorama==0.4.6
comm==0.2.2
contourpy==1.2.1
cryptography==46.0.3
cycler==0.12.1
databricks-sdk==0.73.0
dataclasses-json==0.6.7
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.8
diskcache==5.6.3
distlib==0.4.0
distro==1.9.0
docopt==0.6.2
docstring_parser==0.17.0
evaluate==0.4.6
executing==2.0.1
faiss-cpu==1.10.0
fastapi==0.121.0
fastjsonschema==2.20.0
filelock==3.15.4
fonttools==4.53.1
frozenlist==1.8.0
fsspec==2024.6.1
giskard==2.18.0
gitdb==4.0.12
GitPython==3.1.45
google-auth==2.43.0
greenlet==3.2.4
griffe==1.14.0
groq==0.33.0
h11==0.16.0
hf-xet==1.2.0
httpcore==1.0.9
httpx==0.28.1
httpx-sse==0.4.3
huggingface-hub==0.36.0
identify==2.6.15
idna==3.7
importlib_metadata==8.7.0
instructor==1.13.0
intel-openmp==2023.2.0
ipydatawidgets==4.3.5
ipython==8.12.3
ipyvolume==0.6.3
ipyvue==1.11.1
ipyvuetify==1.10.0
ipywebrtc==0.6.0
ipywidgets==8.1.5
isodate==0.7.2
jedi==0.19.1
Jinja2==3.1.4
jiter==0.11.1
joblib==1.4.2
jsonpatch==1.33
jsonpointer==3.0.0
jsonschema==4.22.0
jsonschema-specifications==2023.12.1
jupyter_client==8.6.2
jupyter_core==5.7.2
jupyterlab_pygments==0.3.0
jupyterlab_widgets==3.0.13
kiwisolver==1.4.5
langchain==1.0.5
langchain-azure-ai==1.0.3
langchain-classic==1.0.0
langchain-community==0.4.1
langchain-core==1.0.4
langchain-openai==1.0.2
langchain-text-splitters==1.0.0
langdetect==1.0.9
langgraph==1.0.2
langgraph-checkpoint==3.0.1
langgraph-prebuilt==1.0.2
langgraph-sdk==0.2.9
langsmith==0.4.39
lazy_loader==0.4
librosa==0.10.1
lightgbm==4.4.0
litellm==1.65.8
llvmlite==0.41.1
Markdown==3.10
markdown-it-py==4.0.0
MarkupSafe==2.1.5
marshmallow==3.26.1
matplotlib==3.7.5
matplotlib-inline==0.1.7
mdurl==0.1.2
mistune==3.0.2
mixpanel==5.0.0
mkl==2023.2.2
mlflow-skinny==3.5.1
mpmath==1.3.0
msal==1.34.0
msal-extensions==1.3.1
msgpack==1.0.8
multidict==6.7.0
multiprocess==0.70.16
mypy_extensions==1.1.0
nbclient==0.10.0
nbconvert==7.16.4
nbformat==5.10.4
nest-asyncio==1.6.0
networkx==3.3
nodeenv==1.9.1
num2words==0.5.14
numba==0.58.1
numpy==1.26.4
openai==2.7.1
opencv-python==4.10.0.84
opentelemetry-api==1.38.0
opentelemetry-proto==1.38.0
opentelemetry-sdk==1.38.0
opentelemetry-semantic-conventions==0.59b0
orjson==3.11.4
ormsgpack==1.12.0
packaging==24.1
pandas==2.0.3
pandocfilters==1.5.1
parso==0.8.4
pathlib==1.0.1
pexpect==4.9.0
pickleshare==0.7.5
pillow==10.4.0
pip==25.3
pipreqs==0.5.0
platformdirs==4.2.2
pooch==1.8.2
pre_commit==4.4.0
prompt_toolkit==3.0.47
propcache==0.4.1
protobuf==6.33.0
ptyprocess==0.7.0
pure-eval==0.2.2
pyasn1==0.6.1
pyasn1_modules==0.4.2
pycparser==2.22
pydantic==2.12.3
pydantic_core==2.41.4
pydantic-settings==2.11.0
pydub==0.25.1
Pygments==2.18.0
PyJWT==2.10.1
pynndescent==0.5.13
pyparsing==3.1.2
pysbd==0.3.4
pytesseract==0.3.10
python-dateutil==2.8.2
python-dotenv==1.2.1
python-toon==0.1.3
pythreejs==2.4.2
pytz==2023.3.post1
PyVTK==0.5.18
PyYAML==6.0.3
pyzmq==26.0.3
ragas==0.3.9
referencing==0.35.1
regex==2025.11.3
requests==2.32.5
requests-toolbelt==1.0.0
rich==14.2.0
rpds-py==0.18.1
rsa==4.9.1
safetensors==0.6.2
scikit-base==0.7.8
scikit-learn==1.7.2
scikit-network==0.33.3
scipy==1.11.4
setuptools==65.5.0
shellingham==1.5.4
six==1.17.0
sktime==0.29.1
smmap==5.0.2
sniffio==1.3.1
soundfile==0.12.1
soupsieve==2.5
soxr==0.3.7
SQLAlchemy==2.0.44
sqlparse==0.5.3
stack-data==0.6.3
starlette==0.49.3
sympy==1.12.1
tbb==2021.10.0
tenacity==9.1.2
threadpoolctl==3.5.0
tiktoken==0.12.0
tinycss2==1.3.0
tokenizers==0.22.1
torch==2.2.2
tornado==6.4.1
tqdm==4.67.1
traitlets==5.14.3
traittypes==0.2.1
transformers==4.57.1
ty==0.0.1a26
typer==0.20.0
typing_extensions==4.15.0
typing-inspect==0.9.0
typing-inspection==0.4.2
tzdata==2023.3
umap-learn==0.5.9.post2
urllib3==2.2.2
uvicorn==0.38.0
virtualenv==20.35.4
wcwidth==0.2.13
webencodings==0.5.1
widgetsnbextension==4.0.13
xxhash==3.6.0
xyzservices==2025.10.0
yarg==0.1.9
yarl==1.22.0
zipp==3.23.0
zstandard==0.25.0
```

### Current Behaviour?

```shell
### Current Behaviour
When running `evaluate()` with **ragas==0.3.9**, Giskard’s RAG evaluation breaks due to upstream API changes in the RAGAS library.

Two issues appear:

1. **Abstract Method Error**  
   `RagasLLMWrapper` cannot be instantiated because `BaseRagasLLM` in RAGAS 0.3.9 adds a new abstract method `is_finished()`.  
   This leads to:  

TypeError: Can't instantiate abstract class RagasLLMWrapper with abstract method is_finished

2. **Metric Scoring Error**

    All RAGAS metrics (e.g., `AnswerRelevancy`) removed `.score(dict)` and replaced it with `single_turn_score(SingleTurnSample)` causing:

AttributeError: 'AnswerRelevancy' object has no attribute 'score'


As a result, the evaluation pipeline fails immediately and **no RAG metrics can be computed** using the latest RAGAS version.

---

### Expected Behaviour

Giskard’s `evaluate()` function should seamlessly work with **RAGAS ≥ 0.3.3**, correctly executing all RAG metrics (Answer Relevancy, Faithfulness, Context Precision, Context Recall, etc.) without errors.

Specifically:

- `RagasLLMWrapper` should instantiate successfully (implementing the required `is_finished()` method).
- Metric scoring should automatically adapt to the new RAGAS API by using:


        sample = SingleTurnSample(...)
        val = metric.single_turn_score(sample)

**Evaluation Reports** should be generated normally with updated metrics.
```

### Standalone code OR list down the steps to reproduce the issue

```shell
## Steps to reproduce

1. Create and activate a Python environment (e.g. Conda or venv).

2. Make sure you have an Azure OpenAI resource (chat + embedding deployments).

3. Set the following environment variables with your actual values:

   - **AZURE_API_BASE** – Azure OpenAI endpoint  
   - **AZURE_API_KEY** – Azure OpenAI API key  
   - **AZURE_API_VERSION** – e.g. 2024-02-01  
   - **AZURE_OPENAI_EMBEDDING_MODEL_DEPLOYMENT_NAME** – embedding deployment name  
   - **AZURE_OPENAI_CHAT_MODEL_EVALUATION_DEPLOYMENT_NAME** – chat deployment name  

4. Prepare a small CSV with columns:  
   `article_title`, `article_text`, `article_recitals`.  
   Or download this example CSV:  
   https://drive.google.com/file/d/1Z0p4JbE9FGCIATD3H-a3WV3pMhy-PQcE/view?usp=drive_link

5. Save the script below as `repro_giskard_ragas.py`:


from giskard.llm import set_llm_model, set_embedding_model
import os
from giskard.rag import evaluate
from giskard.rag.metrics import correctness_metric
from giskard.rag.metrics.ragas_metrics import (
    ragas_faithfulness,
    ragas_context_recall,
    ragas_context_precision,
    ragas_answer_relevancy,
)
import os
import pandas as pd
from giskard.rag import KnowledgeBase
from giskard.rag import QATestset, generate_testset
from giskard.rag import AgentAnswer

# -------------------------------------------------------------------
# Azure OpenAI configuration (replace with real values)
# -------------------------------------------------------------------
os.environ["AZURE_API_BASE"] ="Add azure openai key endpoint"
os.environ["AZURE_API_KEY"] ="add azure openai key"
os.environ["AZURE_API_VERSION"] ="add azure api version"
os.environ["AZURE_OPENAI_EMBEDDING_MODEL_DEPLOYMENT_NAME"] ="add embedding deployment name"
os.environ["AZURE_OPENAI_CHAT_MODEL_EVALUATION_DEPLOYMENT_NAME"] ="add chat deployment name"

set_llm_model(f"azure/{os.environ['AZURE_OPENAI_CHAT_MODEL_EVALUATION_DEPLOYMENT_NAME']}")
set_embedding_model(f"azure/{os.environ['AZURE_OPENAI_EMBEDDING_MODEL_DEPLOYMENT_NAME']}")

CSV_PATH="/gdpr_cased_articles_with_recitals.csv"
df = pd.read_csv(CSV_PATH)

def row_to_text(r):
    parts = []
    if r.get("article_title"):  parts.append(str(r["article_title"]))
    if r.get("article_text"):   parts.append(str(r["article_text"]))
    if r.get("article_recitals"):
        parts.append("Recitals:
" + str(r["article_recitals"]))
    return "\n\n".join(parts)

kb_df = pd.DataFrame({"text": [row_to_text(r) for _, r in df.iterrows()]})
knowledge_base = KnowledgeBase.from_pandas(kb_df, columns=["text"])

testset = generate_testset(
    knowledge_base,
    num_questions=1,
    language="en",
    agent_description="A GDPR compliance bot that answers strictly from GDPR Articles & Recitals.",
)

def answer_fn(question: str) -> AgentAnswer:
    return AgentAnswer(
        message="Dummy GDPR answer",
        documents=["Some dummy retrieved snippet about GDPR"]
    )

metrics = [
    correctness_metric,
    ragas_answer_relevancy,
    ragas_faithfulness,
    ragas_context_precision,
    ragas_context_recall,
]

# -------------------------------------------------------------------
# Run evaluation – triggers:
# TypeError: Can't instantiate abstract class RagasLLMWrapper with abstract method is_finished
# -------------------------------------------------------------------

report = evaluate(
    answer_fn,
    testset=testset,
    knowledge_base=knowledge_base,
    metrics=metrics,
)


6. Run the script:


python repro_giskard_ragas.py


### Expected Error (1):

TypeError: Can't instantiate abstract class RagasLLMWrapper with abstract method is_finished


---

## Solution Attempt

Implement `is_finished` method inside `RagasLLMWrapper`:


from langchain_core.outputs import ChatGeneration

def is_finished(self, response: LLMResult) -> bool:
    flags: list[bool] = []
    for gens in response.generations:
        if not gens:
            flags.append(True)
            continue
        resp = gens[0]
        fin = None

        if getattr(resp, "generation_info", None):
            fin = resp.generation_info.get("finish_reason")

        if fin is None and isinstance(resp, ChatGeneration):
            meta = resp.message.response_metadata or {}
            fin = meta.get("finish_reason") or meta.get("stop_reason")

        flags.append(fin in ["stop", "STOP", "end_turn", "MAX_TOKENS", None])

    return all(flags)


After fixing this, the next error appears:

### Expected Error (2):


AttributeError: 'AnswerRelevancy' object has no attribute 'score'


This happens because **ragas 0.3.x changed the metric API**.

Older API:

metric.score(ragas_sample_dict)


New API (ragas 0.3+):

metric.single_turn_score(SingleTurnSample)


---

## Local Fix Implemented

### Convert `ragas_sample` to `SingleTurnSample`:


sample = SingleTurnSample(
    user_input=ragas_sample.get("user_input") or ragas_sample.get("question"),
    response=ragas_sample.get("response") or ragas_sample.get("answer"),
    retrieved_contexts=ragas_sample.get("retrieved_contexts") or ragas_sample.get("contexts"),
    reference=ragas_sample.get("reference") or ragas_sample.get("ground_truth"),
)


### Use the correct API depending on version:


if hasattr(self.metric, "single_turn_score"):
    val = self.metric.single_turn_score(sample)
else:
    val = self.metric.score(ragas_sample)

return {self.name: val}
```

### Relevant log output

```shell
## First Error Logs: TypeError - RagasLLMWrapper Missing is_finished()

## Full Error Log


TypeError                                 Traceback (most recent call last)
Cell In[1], line 75
     62 metrics = [
     63     correctness_metric,
     64     ragas_answer_relevancy,
   (...)     67     ragas_context_recall,
     68 ]
     70 # -------------------------------------------------------------------
     71 # Run evaluation – triggers:
     72 # TypeError: Can't instantiate abstract class RagasLLMWrapper with abstract method is_finished
     73 # -------------------------------------------------------------------
---> 75 report = evaluate(
     76     answer_fn,
     77     testset=testset,
     78     knowledge_base=knowledge_base,
     79     metrics=metrics,
     80 )

File /opt/anaconda3/envs/RAG_Chat_Bot/lib/python3.11/site-packages/giskard/rag/evaluate.py:105, in evaluate(answer_fn, testset, knowledge_base, llm_client, agent_description, metrics)
     98         metric_name = metric.__name__
    100     for sample, answer in maybe_tqdm(
    101         zip(testset.to_pandas().to_records(index=True), model_outputs),
    102         desc=f"{metric_name} evaluation",
    103         total=len(model_outputs),
    104     ):
--> 105         metrics_results[sample["id"]].update(metric(sample, answer))
    107 report = RAGReport(testset, model_outputs, metrics_results, knowledge_base)
    108 recommendation = get_rag_recommendation(
    109     report.topics,
    110     report.correctness_by_question_type().to_dict()[metrics[0].name],
    111     report.correctness_by_topic().to_dict()[metrics[0].name],
    112     llm_client,
    113 )

File /opt/anaconda3/envs/RAG_Chat_Bot/lib/python3.11/site-packages/giskard/rag/metrics/ragas_metrics.py:159, in RagasMetric.__call__(self, question_sample, answer)
    157 embedding_model = self._embedding_model or get_default_embedding()
    158 if self.ragas_llm is None:
--> 159     self.ragas_llm = RagasLLMWrapper(llm_client, self.context_window_length)
    160 if self.ragas_embeddings is None:
    161     self.ragas_embeddings = RagasEmbeddingsWrapper(embedding_model)

TypeError: Can't instantiate abstract class RagasLLMWrapper with abstract method is_finished


---

## Second Error: AttributeError: 'AnswerRelevancy' object has no attribute 'score'

## Full Error Log


AttributeError                            Traceback (most recent call last)
Cell In[1], line 75
     62 metrics = [
     63     correctness_metric,
     64     ragas_answer_relevancy,
   (...)     67     ragas_context_recall,
     68 ]
     70 # -------------------------------------------------------------------
     71 # Run evaluation – triggers:
     72 # TypeError: Can't instantiate abstract class RagasLLMWrapper with abstract method is_finished
     73 # -------------------------------------------------------------------
---> 75 report = evaluate(
     76     answer_fn,
     77     testset=testset,
     78     knowledge_base=knowledge_base,
     79     metrics=metrics,
     80 )

File /opt/anaconda3/envs/RAG_Chat_Bot/lib/python3.11/site-packages/giskard/rag/evaluate.py:105, in evaluate(answer_fn, testset, knowledge_base, llm_client, agent_description, metrics)
     98         metric_name = metric.__name__
    100     for sample, answer in maybe_tqdm(
    101         zip(testset.to_pandas().to_records(index=True), model_outputs),
    102         desc=f"{metric_name} evaluation",
    103         total=len(model_outputs),
    104     ):
--> 105         metrics_results[sample["id"]].update(metric(sample, answer))
    107 report = RAGReport(testset, model_outputs, metrics_results, knowledge_base)
    108 recommendation = get_rag_recommendation(
    109     report.topics,
    110     report.correctness_by_question_type().to_dict()[metrics[0].name],
    111     report.correctness_by_topic().to_dict()[metrics[0].name],
    112     llm_client,
    113 )

File /opt/anaconda3/envs/RAG_Chat_Bot/lib/python3.11/site-packages/giskard/rag/metrics/ragas_metrics.py:180, in RagasMetric.__call__(self, question_sample, answer)
    176     return {self.name: 0}
    178 ragas_sample = self.prepare_ragas_sample(question_sample, answer)
--> 180 return {self.name: self.metric.score(ragas_sample)}

AttributeError: 'AnswerRelevancy' object has no attribute 'score'


---
```