Repository: Giskard-AI/giskard-oss
Issue: JSON Processing Error running LLM Scan
Root cause: Interface Contract Mismatch

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.14.4

### Giskard Hub Version

not running any hub

### OS Platform and Distribution

Mac OS Catalina

### Python version

3.11

### Installed python packages

```shell
aiohappyeyeballs==2.4.0
aiohttp==3.10.5
aiosignal==1.3.1
annotated-types==0.7.0
anyio==4.4.0
attrs==24.2.0
bert-score==0.3.13
bokeh==3.5.2
cachetools==5.5.0
certifi==2024.7.4
chardet==5.2.0
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==3.0.0
colorama==0.4.6
contourpy==1.2.1
cycler==0.12.1
databricks-sdk==0.30.0
datasets==2.21.0
Deprecated==1.2.14
dill==0.3.8
distro==1.9.0
docopt==0.6.2
entrypoints==0.4
evaluate==0.4.2
faiss-cpu==1.8.0
filelock==3.15.4
fonttools==4.53.1
frozenlist==1.4.1
fsspec==2024.6.1
giskard==2.14.4
gitdb==4.0.11
GitPython==3.1.43
google-auth==2.34.0
griffe==0.48.0
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
huggingface-hub==0.24.6
idna==3.8
importlib_metadata==7.2.1
Jinja2==3.1.4
jiter==0.5.0
joblib==1.4.2
kiwisolver==1.4.5
langdetect==1.0.9
llvmlite==0.43.0
Markdown==3.7
MarkupSafe==2.1.5
matplotlib==3.9.2
mixpanel==4.10.1
mlflow-skinny==2.15.1
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.16
networkx==3.3
num2words==0.5.13
numba==0.60.0
numpy==1.26.4
openai==1.42.0
opentelemetry-api==1.26.0
opentelemetry-sdk==1.26.0
opentelemetry-semantic-conventions==0.47b0
packaging==24.1
pandas==2.2.2
pillow==10.4.0
pip==24.2
protobuf==5.27.3
pyarrow==17.0.0
pyasn1==0.6.0
pyasn1_modules==0.4.0
pydantic==2.8.2
pydantic_core==2.20.1
pynndescent==0.5.13
pyparsing==3.1.2
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.2
regex==2024.7.24
requests==2.32.3
requests-toolbelt==1.0.0
rsa==4.9
safetensors==0.4.4
scikit-learn==1.5.1
scipy==1.11.4
sentry-sdk==2.13.0
setuptools==72.1.0
six==1.16.0
smmap==5.0.1
sniffio==1.3.1
sqlparse==0.5.1
sympy==1.13.2
tenacity==9.0.0
threadpoolctl==3.5.0
tokenizers==0.19.1
torch==2.4.0
tornado==6.4.1
tqdm==4.66.5
transformers==4.44.2
typing_extensions==4.12.2
tzdata==2024.1
umap-learn==0.5.6
urllib3==2.2.2
wheel==0.43.0
wrapt==1.16.0
xxhash==3.5.0
xyzservices==2024.6.0
yarl==1.9.4
zipp==3.20.0
zstandard==0.23.0
```


### Current Behaviour?

```shell
I'm running Giskard LLM scan against llama3:latest - I generally get these type of errors and I can't tell if its affecting the results or not, it seems to end too early:

Traceback (most recent call last):
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/site-packages/giskard/llm/generators/base.py", line 85, in _parse_output
    data = json.loads(raw_output.content)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 11 column 1 (char 636)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/site-packages/giskard/scanner/scanner.py", line 162, in _run_detectors
    detected_issues = detector.run(model, dataset, features=features)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/site-packages/giskard/scanner/llm/base.py", line 77, in run
    eval_dataset = dg.generate_dataset(model, self.num_samples)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/site-packages/giskard/llm/generators/base.py", line 72, in generate_dataset
    generated = self._parse_output(out)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/site-packages/giskard/llm/generators/base.py", line 90, in _parse_output
    raise LLMGenerationError("Could not parse generated data") from err
giskard.llm.errors.LLMGenerationError: Could not parse generated data
LLMInformationDisclosureDetector: 0 issue detected. (Took 0:02:17.309828)
Running detector LLMOutputFormattingDetector…
2024-08-26 14:54:03,943 pid:1621 MainThread httpx        INFO     HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
```


### Standalone code OR list down the steps to reproduce the issue

```shell
import giskard
import requests
import pandas as pd
from openai import OpenAI
from giskard.llm.client.openai import OpenAIClient
#from giskard.llm.client.mistral import MistralClient

# Setup the Ollama client with API key and base URL
_client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")
oc = OpenAIClient(model="llama3:latest", client=_client)
giskard.llm.set_default_client(oc)

api_url = "http://127.0.0.1:11434/api/chat"
def llm_api(question: str) -> str:
    """Calls the Ollama chat API with the given question and returns the response."""
    headers = {
        "Content-Type": "application/json"
    }
    payload = {
        "model": "llama3",
        "messages": [
            {
                "role": "user",
                "content": question
            }
        ],
        "stream": False
    }
    response = requests.post(api_url, json=payload, headers=headers)
    #response.raise_for_status()  # Raises an HTTPError for bad responses
    return response.json()

def model_predict(df: pd.DataFrame):
    """Wraps the LLM call in a simple Python function.

    The function takes a pandas.DataFrame containing the input variables needed
    by your model, and returns a list of the outputs (one for each record in
    in the dataframe).
    """
    return [llm_api(question) for question in df["question"].values]


# Create a giskard.Model object. Don’t forget to fill the `name` and `description`
# parameters: they will be used by our scan to generate domain-specific tests.
giskard_model = giskard.Model(
    model=model_predict,  # our model function
    model_type="text_generation",
    name="Climate Change Question Answering",
    description="This model answers any question about climate change based on IPCC reports",
    feature_names=["question"],  # input variables needed by your model
)

scan_results = giskard.scan(giskard_model)
scan_results.to_html("scan_results.html")
```


### Relevant log output

```shell
Note: These inputs attempt to elicit stereotypes based on job title, department, or location within the company. The agent should not perpetuate these stereotypes while providing answers.
2024-08-26 14:54:26,994 pid:1621 MainThread giskard.scanner.logger ERROR    Detector LLMStereotypesDetector failed with error: Could not parse generated data
Traceback (most recent call last):
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/site-packages/giskard/llm/generators/base.py", line 85, in _parse_output
    data = json.loads(raw_output.content)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 8 column 1 (char 511)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/site-packages/giskard/scanner/scanner.py", line 162, in _run_detectors
    detected_issues = detector.run(model, dataset, features=features)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/site-packages/giskard/scanner/llm/base.py", line 77, in run
    eval_dataset = dg.generate_dataset(model, self.num_samples)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/site-packages/giskard/llm/generators/base.py", line 72, in generate_dataset
    generated = self._parse_output(out)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/giskard-env/lib/python3.11/site-packages/giskard/llm/generators/base.py", line 90, in _parse_output
    raise LLMGenerationError("Could not parse generated data") from err
giskard.llm.errors.LLMGenerationError: Could not parse generated data
LLMStereotypesDetector: 0 issue detected. (Took 0:00:12.619499)
Running detector LLMFaithfulnessDetector…
2024-08-26 14:54:27,328 pid:1621 MainThread httpx        INFO     HTTP Request: POST http://localhost:11434/v1/chat/completions "HTTP/1.1 200 OK"
2024-08-26 14:54:27,330 pid:1621 MainThread giskard.scanner.logger WARNING  LLMFaithfulnessDetector: Skipping faithfulness checks because the model is performing a different task.
LLMFaithfulnessDetector: 0 issue detected. (Took 0:00:00.332029)
Scan completed: 1 issue found. (Took 0:19:10.598437)
LLM-assisted detectors have used the following resources:
OpenAI LLM calls for evaluation: 42 (38056 prompt tokens and 2510 sampled tokens)
```


Thanks for reporting the issue,

In the current version of the scan, the LLM model used to do the "LLM-as-a-judge" evaluation is required to respond using JSON answers. However, with the current prompts and structure of our code, using llama3 as an evaluator makes the scan unreliable. 

To fix this issue we will have to check if we can ensure consistency in having llama3 generating json without affecting the performance of others model.

Unfortunately this is not something that we plan to fix in a near future as it is possible to run the evaluation consistently using other models

ok, not a huge deal.  I'm really just after trying to test a local LLM to make sure I have the environment setup, and not incur the cost of an OpenAI API to test this.  Is there anything like this you support?