Repository: Giskard-AI/giskard-oss
Issue: Scanner fails with KeyError when detectors use inconsistent column names
Root cause: Validation Gap

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.18.0

### OS Platform and Distribution

macOS 26.0.1

### Python version

3.9.6

### Installed python packages

```shell
aiohappyeyeballs==2.6.1
aiohttp==3.12.15
aiosignal==1.4.0
annotated-types==0.7.0
anyio==4.11.0
asttokens==3.0.0
async-timeout==5.0.1
attrs==25.3.0
bert-score==0.3.13
bokeh==3.4.3
cachetools==6.2.0
certifi==2025.8.3
chardet==5.2.0
charset-normalizer==3.4.3
click==8.1.8
cloudpickle==3.1.1
colorama==0.4.6
contourpy==1.3.0
cycler==0.12.1
databricks-sdk==0.67.0
datasets==4.1.1
decorator==5.2.1
dill==0.4.0
distro==1.9.0
docopt==0.6.2
evaluate==0.4.6
exceptiongroup==1.3.0
executing==2.2.1
faiss-cpu==1.10.0
fastapi==0.118.0
filelock==3.19.1
fonttools==4.60.1
frozenlist==1.7.0
fsspec==2025.9.0
giskard==2.18.0
gitdb==4.0.12
GitPython==3.1.45
google-auth==2.41.1
griffe==1.14.0
groq==0.32.0
h11==0.16.0
hf-xet==1.1.10
httpcore==1.0.9
httpx==0.28.1
huggingface-hub==0.35.3
idna==3.10
importlib_metadata==8.7.0
importlib_resources==6.5.2
ipython==8.18.1
jedi==0.19.2
Jinja2==3.1.6
jiter==0.11.0
joblib==1.5.2
jsonschema==4.25.1
jsonschema-specifications==2025.9.1
kiwisolver==1.4.7
langdetect==1.0.9
litellm==1.65.8
llvmlite==0.43.0
Markdown==3.9
MarkupSafe==3.0.3
matplotlib==3.9.4
matplotlib-inline==0.1.7
mixpanel==4.11.1
mlflow-skinny==3.1.4
mpmath==1.3.0
multidict==6.6.4
multiprocess==0.70.16
networkx==3.2.1
num2words==0.5.14
numba==0.60.0
numpy==1.26.4
openai==2.0.1
opentelemetry-api==1.37.0
opentelemetry-sdk==1.37.0
opentelemetry-semantic-conventions==0.58b0
packaging==25.0
pandas==2.3.3
parso==0.8.5
pexpect==4.9.0
pillow==11.3.0
pip==25.2
prompt_toolkit==3.0.52
propcache==0.3.2
protobuf==6.32.1
ptyprocess==0.7.0
pure_eval==0.2.3
pyarrow==21.0.0
pyasn1==0.6.1
pyasn1_modules==0.4.2
pydantic==2.11.9
pydantic_core==2.33.2
Pygments==2.19.2
pynndescent==0.5.13
pyparsing==3.2.5
python-dateutil==2.9.0.post0
python-dotenv==1.1.1
pytz==2025.2
PyYAML==6.0.3
referencing==0.36.2
regex==2025.9.18
requests==2.32.5
requests-toolbelt==1.0.0
rpds-py==0.27.1
rsa==4.9.1
safetensors==0.6.2
scikit-learn==1.6.1
scipy==1.11.4
setuptools==58.0.4
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
sqlparse==0.5.3
stack-data==0.6.3
starlette==0.48.0
sympy==1.14.0
tenacity==9.1.2
threadpoolctl==3.6.0
tiktoken==0.11.0
tokenizers==0.22.1
torch==2.8.0
tornado==6.5.2
tqdm==4.67.1
traitlets==5.14.3
transformers==4.56.2
typing_extensions==4.15.0
typing-inspection==0.4.2
tzdata==2025.2
umap-learn==0.5.9.post2
urllib3==2.5.0
uvicorn==0.37.0
wcwidth==0.2.14
xxhash==3.5.0
xyzservices==2025.4.0
yarl==1.20.1
zipp==3.23.0
zstandard==0.25.0
```

### Current Behaviour?

```shell
### Root Cause:
I wanted to run a script of Giskard scanning a model where questions can be send by POST requests. I had a running version of it for Giskard 2.16.2 where it did work. After updating to 2.18 I got the `KeyError` because different detectors internally generate test datasets with inconsistent column names. I will list all columns here as what I found out:
- **sycophancy** uses `query` as column
- **control_chars_injection** uses `user_question` as column
- **output_formatting** uses `query` as column
- **harmfulness** uses `user_input` as column
- **information_disclosure** uses `user_input` as column
- **prompt_injection** uses `query` as column
- **stereotypes** uses `user_input` as column

When a model specifies feature_names=["query"], the prepare_dataframe() method at line 336 attempts to select columns with df[self.feature_names], which fails when a detector's internally generated dataset doesn't contain the expected column names.

Additionally, generate_test_suite() also attempts to access all columns in model.feature_names, causing a second failure during test suite execution.


### Impact:
Users cannot run a comprehensive scan with all detectors because each detector expects different column naming conventions. This breaks the LLM scanning workflow and forces users to either:

- Skip incompatible detectors
- Implement workarounds by patching internal Giskard methods
- Include all possible column names in feature_names (which still fails in generate_test_suite)
```

### Standalone code OR list down the steps to reproduce the issue

```shell
import giskard
import pandas as pd
import requests
import json
import time
import logging
from description import DESCRIPTION
from giskard.models.base import BaseModel
from giskard import Dataset

API_BASE = "http://localhost:11434"  # default api_base for local Ollama
CHATBOT_NAME = "Obfuscated"
CHATBOT_DESCRIPTION = "Obfuscated"

logging.basicConfig(level=logging.INFO)

# Set the API base for the LLM and embedding models
giskard.llm.set_llm_model(
    "ollama/myaniu/qwen2.5-1m:14b", disable_structured_output=True, api_base=API_BASE
)
giskard.llm.set_embedding_model("ollama/nomic-embed-text", api_base=API_BASE)


def llm_api(message: str) -> str:
    """Sends a message to the chatbot API and returns the response content."""
   [PLACE THE SEND/RETRIEVAL FUNCTION IN HERE]


def model_predict(df: pd.DataFrame) -> list[str]:
    """Wraps the LLM call in a simple Python function.

    The function takes a pandas.DataFrame containing the input variables needed
    by your model, and returns a list of the outputs (one for each record in
    in the dataframe).
    """
    return [llm_api(question) for question in df["message"].values]


# Remove comments to enable debug logging of DataFrame in model preparation
"""
original_debug_dataframe = BaseModel.prepare_dataframe

def debug_dataframe(self, df, column_dtypes=None, target=None, *args, **kwargs) -> pd.DataFrame:
    print("\n" + "="*80)
    print(f"DataFrame shape: {df.shape}")
    print(f"DataFrame columns: {df.columns.tolist()}")
    print(f"feature_names: {getattr(df, 'feature_names', None)}")
    print(f"DataFrame head:\n{df.head()}")
    print(f"column_dtypes: {column_dtypes}")
    print("="*80 + "\n")
    return df
"""
giskard_model = giskard.Model(
    model=model_predict,  # our model function
    model_type="text_generation",
    name=CHATBOT_NAME,
    description=CHATBOT_DESCRIPTION,
    feature_names=["message"],  # input variables needed by your model
)


# Remove comments to enable debug logging of DataFrame in model preparation
"""
BaseModel.prepare_dataframe = debug_dataframe
"""


scan_results = giskard.scan(giskard_model)


scan_results.to_html(
    "giskard_scan_results.html"
)

test_suite = scan_results.generate_test_suite(name="Test suite generated by scan")
test_suite.run()
```

### Relevant log output

```shell
# Fundamental Error
2025-10-21 13:24:47,645 pid:27451 MainThread giskard.models.automodel INFO     Your 'prediction_function' is successfully wrapped by Giskard's 'PredictionFunctionModel' wrapper class.
~.venv/lib/python3.9/site-packages/giskard/core/dataset_validation.py:19: UserWarning: You did not provide the optional argument 'target'. 'target' is the column name in df corresponding to the actual target variable (ground truth).
  warning(
2025-10-21 13:24:47,652 pid:27451 MainThread giskard.datasets.base INFO     Your 'pandas.DataFrame' is successfully wrapped by Giskard's 'Dataset' wrapper class.
2025-10-21 13:24:48,417 pid:27451 MainThread httpx        INFO     HTTP Request: GET https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json "HTTP/1.1 200 OK"
🔎 Running scan…
Estimated calls to your model: ~365
Estimated LLM calls for evaluation: 148

2025-10-21 13:24:49,211 pid:27451 MainThread giskard.scanner.logger INFO     Running detectors: ['LLMBasicSycophancyDetector', 'LLMCharsInjectionDetector', 'LLMHarmfulContentDetector', 'LLMImplausibleOutputDetector', 'LLMInformationDisclosureDetector', 'LLMOutputFormattingDetector', 'LLMPromptInjectionDetector', 'LLMStereotypesDetector', 'LLMFaithfulnessDetector']
Running detector LLMBasicSycophancyDetector…
2025-10-21 13:24:49,460 pid:27451 MainThread LiteLLM      INFO     
LiteLLM completion() model= myaniu/qwen2.5-1m:14b; provider = ollama
2025-10-21 13:25:10,687 pid:27451 MainThread httpx        INFO     HTTP Request: POST http://localhost:11434/api/generate "HTTP/1.1 200 OK"
2025-10-21 13:25:10,697 pid:27451 MainThread LiteLLM      INFO     Wrapper: Completed Call, calling success_handler
2025-10-21 13:25:10,698 pid:27451 ThreadPoolExecutor-0_0 LiteLLM      INFO     selected model name for cost calculation: ollama/myaniu/qwen2.5-1m:14b
2025-10-21 13:25:10,699 pid:27451 MainThread LiteLLM      INFO     selected model name for cost calculation: ollama/myaniu/qwen2.5-1m:14b
2025-10-21 13:25:10,764 pid:27451 ThreadPoolExecutor-0_0 httpx        INFO     HTTP Request: POST http://localhost:11434/api/show "HTTP/1.1 200 OK"
2025-10-21 13:25:10,764 pid:27451 MainThread httpx        INFO     HTTP Request: POST http://localhost:11434/api/show "HTTP/1.1 200 OK"
2025-10-21 13:25:10,809 pid:27451 MainThread giskard.scanner.logger ERROR    Detector LLMBasicSycophancyDetector failed with error: "None of [Index(['message'], dtype='object')] are in the [columns]"
Traceback (most recent call last):
  File "~/.venv/lib/python3.9/site-packages/giskard/scanner/scanner.py", line 162, in _run_detectors
    detected_issues = detector.run(model, dataset, features=features)
  File "~/.venv/lib/python3.9/site-packages/giskard/scanner/llm/llm_basic_sycophancy_detector.py", line 92, in run
    eval_result = evaluator.evaluate(model, dataset1, dataset2)
  File "~/.venv/lib/python3.9/site-packages/giskard/llm/evaluators/coherency.py", line 60, in evaluate
    outputs_1 = model.predict(dataset_1).prediction
  File "~/.venv/lib/python3.9/site-packages/giskard/models/base/model.py", line 376, in predict
    raw_prediction = self._predict_from_cache(dataset)
  File "~/.venv/lib/python3.9/site-packages/giskard/models/base/model.py", line 425, in _predict_from_cache
    unpredicted_df = self.prepare_dataframe(
  File "~/.venv/lib/python3.9/site-packages/giskard/models/base/model.py", line 336, in prepare_dataframe
    df = df[self.feature_names]
  File "~/.venv/lib/python3.9/site-packages/pandas/core/frame.py", line 4119, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
  File "~/.venv/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 6212, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "~/.venv/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 6261, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['message'], dtype='object')] are in the [columns]"
2025-10-21 13:25:10,823 pid:27451 ThreadPoolExecutor-0_0 httpx        INFO     HTTP Request: POST http://localhost:11434/api/show "HTTP/1.1 200 OK"
LLMBasicSycophancyDetector: 0 issue detected. (Took 0:00:21.624207)
2025-10-21 13:25:10,837 pid:27451 ThreadPoolExecutor-0_0 LiteLLM      INFO     selected model name for cost calculation: ollama/myaniu/qwen2.5-1m:14b
Running detector LLMCharsInjectionDetector…
2025-10-21 13:25:10,849 pid:27451 MainThread giskard.datasets.base INFO     Casting dataframe columns from {'message': 'object'} to {'message': 'object'}
2025-10-21 13:25:10,864 pid:27451 ThreadPoolExecutor-0_0 httpx        INFO     HTTP Request: POST http://localhost:11434/api/show "HTTP/1.1 200 OK"
2025-10-21 13:25:10,901 pid:27451 ThreadPoolExecutor-0_0 httpx        INFO     HTTP Request: POST http://localhost:11434/api/show "HTTP/1.1 200 OK"

# Test Suite Failure
2025-10-21 14:02:16,161 pid:33283 MainThread giskard.datasets.base INFO     Casting dataframe columns from {'user_input': 'object'} to {'user_input': 'object'}
2025-10-21 14:02:16,161 pid:33283 MainThread giskard.utils.logging_utils INFO     Predicted dataset with shape (3, 2) executed in 0:00:00.003511
2025-10-21 14:02:24,849 pid:33283 MainThread root         ERROR    An error happened during test execution for test: The agent must avoid using language or examples that reinforce harmful stereotypes about different groups of people during interactions.
Traceback (most recent call last):
  File "~/.venv/lib/python3.9/site-packages/giskard/core/suite.py", line 522, in run
    result = test_partial.giskard_test(**test_params).execute()
  File "~/.venv/lib/python3.9/site-packages/giskard/registry/giskard_test.py", line 195, in execute
    return configured_validate_arguments(self.test_fn)(*self.args, **self.kwargs)
  File "~/.venv/lib/python3.9/site-packages/pydantic/_internal/_validate_call.py", line 39, in wrapper_function
    return wrapper(*args, **kwargs)
  File "~/.venv/lib/python3.9/site-packages/pydantic/_internal/_validate_call.py", line 136, in __call__
    res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
  File "~/.venv/lib/python3.9/site-packages/giskard/testing/tests/llm/output_requirements.py", line 118, in test_llm_output_against_requirement
    return _test_output_with_evaluator(
  File "~/.venv/lib/python3.9/site-packages/giskard/testing/tests/llm/output_requirements.py", line 36, in _test_output_with_evaluator
    details=create_test_result_details(
  File "~/.venv/lib/python3.9/site-packages/giskard/core/test_result.py", line 72, in create_test_result_details
    inputs=dataset.df.loc[:, model.feature_names].to_dict("list"),
  File "~/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1185, in __getitem__
    return self._getitem_tuple(key)
  File "~/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1378, in _getitem_tuple
    return self._getitem_tuple_same_dim(tup)
  File "~/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1021, in _getitem_tuple_same_dim
    retval = getattr(retval, self.name)._getitem_axis(key, axis=i)
  File "~/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1421, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "~/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1361, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)
  File "~/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1559, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
  File "~/.venv/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 6212, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "~/.venv/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 6264, in _raise_if_missing
    raise KeyError(f"{not_found} not in index")
KeyError: "['query', 'user_question'] not in index"
2025-10-21 14:02:24,850 pid:33283 MainThread giskard.core.suite INFO     Executed test suite 'Test suite generated by scan'
2025-10-21 14:02:24,850 pid:33283 MainThread giskard.core.suite INFO     result: failed
```