zhiminy/EvalEng huggingface dataset:
Repository: Giskard-AI/giskard-oss
Issue: BUG: Griffe package breaks Giskard
Root cause: External Dependency Breakage

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.14.3

### Giskard Hub Version

NA

### OS Platform and Distribution

Windows 11

### Python version

3.10

### Installed python packages

_No response_

### Current Behaviour?

```shell
Griffe released a new version as of 8/15 - yesterday.  The Giskard install automatically grabs latest version of Griffe which breaks it.  Something to do with docstrings module not found.

Griffe changelog here:
https://mkdocstrings.github.io/griffe/changelog/

I'm guessing this is the breaking change? All objects are available in the top-level griffe module, nowhere else

To resolve:
pip uninstall Griffe
pip install griffe==0.49.0
```


### Standalone code OR list down the steps to reproduce the issue

```shell
To reproduce the issue:
1. create a new python environment
2. pip install giskard
3. try to import giskard
```


### Relevant log output

_No response_

Thanks for reporting the issue, we just release the version 2.14.4 that fixes this
Repository: Giskard-AI/giskard-oss
Issue: Add support for python 3.12
Root cause: Environment Incompatibility

## 🚀 Feature Request
Giskard currently supports up till python 3.11. This request is to extend support for python 3.12.

## 🔈 Motivation

Python 3.12 has had a number of significant improvements over python 3.11 especially on the asyncio front. [Python 3.12 improvements](https://docs.python.org/3/whatsnew/3.12.html)

I'm not sure about everyone else but I've been using python 3.12 since its launch and have been very happy with its performance because LLM applications must run asynchronously in live production. I hope that Giskard will consider extending support for python 3.12.

Hello, this should be available soon, we are testing it to make sure everything works fine

I close this issue since we have another one for #1976
Repository: Giskard-AI/giskard-oss
Issue: Giskard artifacts Pypi issue - local data injection files are not present.
Root cause: Architectural Constraint

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.14.4

### Giskard Hub Version

2.14.4

### OS Platform and Distribution

_No response_

### Python version

_No response_

### Installed python packages

```shell
2.14.4
```


### Current Behaviour?

```shell
When you try to get the latest version of Giskard (2.14.4) on Pypi some files are missing.
e.g. The static files coming from this PR https://github.com/Giskard-AI/giskard/pull/1982 are not visible in the distrib.
```


### Standalone code OR list down the steps to reproduce the issue

```shell
The 'local' folder under giskard-2.14.4\giskard\llm\loaders is not present when installing Giskard through Pypi.
The 'local' folder should contain the both files: giskard_metadata and prompt_injections.
```


### Relevant log output

_No response_

hello @sturiot,
are you still facing this issue? It seems to be fixed since August 23rd: https://github.com/Giskard-AI/giskard/pull/2005
Repository: Giskard-AI/giskard-oss
Issue: Testset generation lost when a single generation fail
Root cause: Architectural Constraint

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.15.0

### OS Platform and Distribution

Macos

### Python version

3.12.4

### Installed python packages

_No response_

### Current Behaviour?

```shell
When generating a testset, it takes around 45-60 minutes to go thru a doc and produce the topics. 

Then it starts generating the testset but... it's failing around 80% because of JSONDecode error and I have to restart again, including all the topic generation part.

We should implement a solution to save the progress, since it is very likely to have error like this when generating testset from very long docs.
```


### Standalone code OR list down the steps to reproduce the issue

```shell
TODO
```


### Relevant log output

_No response_

Thanks for registering this issue. I've seen this behavior happening in a lot of projects when creating large datasets, assuming that the main process won't fail.

Besides catching the error and ignoring those to have the process finish, I think it could be very useful to append the created results to a local file, so that if the process fails somehow, you have some output as a result plus it can be easier to pinpoint the problem in the input data. Otherwise, it just stays in memory and when it fails... it's gone.

Hey @elsatch, thanks a lot for your suggestion.
Would you be interested in implementing this?

Hi @elsatch , just following up on this :)

Hi team,

I've reproduced the problem and agree it's a big pain point—if testset generation fails on a long document, all progress is lost and you have to start over, which wastes a lot of time.

**Suggestion:**  
Would it be possible to add checkpointing or progress saving during testset generation? For example, saving intermediate results after each topic or batch of questions so that, in case of a failure (like a JSONDecode error), the process can resume from the last save instead of starting from scratch.

Also, better error handling and user feedback would help—maybe catching exceptions and logging them, then continuing with the rest of the testset if possible.

If you’re open to it, I’d be happy to help sketch out or prototype these improvements. Let me know what you think!

Thanks everyone!
Repository: Giskard-AI/giskard-oss
Issue: When will the API documentation for the RAG Evaluation Toolkit be completed?
GitHub labels: documentation

question

## Checklist

<!-- Mark with an `x` all the checkboxes that apply (like `[x]`) -->

- [x] I've searched the project's [`issues`](https://github.com/Giskard-AI/giskard/issues?q=is%3Aissue).

## ❓ Question

<!-- What is your question -->

At the moment all articles under this URL https://docs.giskard.ai/en/stable/reference/rag-toolset/index.html are empty. When can we expect to get documentation for the API of the RAGET?



@kevinmessiaen, is it related to #2037 ?

> @kevinmessiaen, is it related to #2037 ?

Yes it is relates but not fixed yet. I need to find you what is making those page empty when building the doc for production. When building for dev locally those page are well populated.

Hey @LordSyd, could you check again the docs webpage? We have fixed it.
https://docs.giskard.ai/en/latest/reference/rag-toolset/index.html
Repository: Giskard-AI/giskard-oss
Issue: Ollama Embeddings
Root cause: Documentation Deficiency

At the moment, the article here https://docs.giskard.ai/en/latest/open_source/setting_up/index.html doesn't mention about how to set up the Ollama client for embedding models. Is this possible or not? If it is, can anyone help show me how?

Hello @nobad4rt, thanks for opening this issue.

Indeed, we should also add the code of how to set up the client for embedding models.
I just opened an internal card so we can update the docs as soon as possible.

In the meanwhile, you can use this code snippet as a base on how to do that:

```python
from openai import OpenAI
from giskard.llm.client.openai import OpenAIClient
from giskard.llm.embeddings.openai import OpenAIEmbedding

_client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")

llm_client = OpenAIClient(model="llama3.2", client=_client)
emb_client = OpenAIEmbedding(model="nomic-embed-text", client=_client)

from giskard.llm import set_default_client
from giskard.llm.embeddings import set_default_embedding

set_default_client(llm_client)
set_default_embedding(emb_client)
```
Repository: Giskard-AI/giskard-oss
Issue: Allow user to specify N retries 
Root cause: Unimplemented Feature Gap

## 🚀 Feature Request

Allow for N (user defined) number of retries of the same question.  

## 🔈 Motivation

In some cases, asking the exact same question 5+ times causes the jailbreak to actually work. Sometimes injection only happens after N retries, like llama3 - I had to insert/ask the same question 5 times and finally it jailbroke and told me how to make a bomb or something like that.  If I was testing with Giskard, it may just test once and call it a pass.

## 🛰 Alternatives

None

## 📎 Additional context

None


Hi @ClarkKentIsSuperman, this is a great idea. Would you be able to take a look at a PR? We can help you get up and running.

seems related https://github.com/Giskard-AI/giskard/issues/2022

Sorry, I don't have time at the moment, but if you can point to where in the code this might be useful (probably needs scorer changes as well) I could start to review where to make the changes?


Is this open to work on?

I think it would be a great feature to add
Repository: Giskard-AI/giskard-oss
Issue: Non-ASCII characters in testsets and report JSON files
Root cause: Configuration Error

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.15.1

### OS Platform and Distribution

Ubuntu 22.04

### Python version

3.12

### Installed python packages

_No response_

### Current Behaviour?

```shell
Right now, the testsets are generated with ASCII characters in the texts. This is not optimal with content in Swedish, for instance.
```


### Standalone code OR list down the steps to reproduce the issue

```shell
Generate a testset in Swedish and/or a report.
```


### Relevant log output

_No response_

My suggestion is to add `force_ascii=False` in these two calls to Pandas `to_json()` function: [testset](https://github.com/Giskard-AI/giskard/blob/750db5c8f87193118d64d71ccbfe513e5132ecb7/giskard/rag/testset.py#L99), [report](https://github.com/Giskard-AI/giskard/blob/750db5c8f87193118d64d71ccbfe513e5132ecb7/giskard/rag/report.py#L161). I can send a PR if you agree.

Hey @PierreMesure, thanks for reporting this issue.
Yes, of course, you can open a PR and request my review

I did it, let me know if you want a test as well. It might take more time.

Thanks @PierreMesure!
Yes, please. Could you add a simple test, for example saving and loading a testset in Swedish, to make sure that the content of the loaded one is similar to the original one?
Thanks a lot!

I'll try! I also noticed that non ASCII-characters end up in other JSON files when generated with [json.dumps](https://github.com/Giskard-AI/giskard/blob/750db5c8f87193118d64d71ccbfe513e5132ecb7/giskard/rag/report.py#L166) so I'll fix it as well.

Thanks a lot @PierreMesure 

Done now. I only added `ensure_ascii=False` in [report.py](https://github.com/Giskard-AI/giskard/blob/750db5c8f87193118d64d71ccbfe513e5132ecb7/giskard/rag/report.py) so you might want to add it in other places where JSON files are saved. But I didn't want to create too much disruption.

I also added a test inspired by another one I found in the same test file but I couldn't get Pytest to work so I don't know if it works. 🤔

EDIT: The test checks that exporting and reimporting a testset doesn't change its content but I don't think that was the case before. The problem I'm trying to solve is the files being hard to read (when opened outside Giskard) because of Unicode characters.
Repository: Giskard-AI/giskard-oss
Issue: Import error with RAGAS v0.2.2
Root cause: External Dependency Breakage

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.15.2

### OS Platform and Distribution

Ubuntu 22.04

### Python version

3.13

### Installed python packages

```shell
ragas==0.2.2
```


### Current Behaviour?

RAGAS just [removed the file ragas.llms.prompt.py](https://github.com/explodinggradients/ragas/pull/1535) from their codebase in their latest minor release (v0.2.2) and this is breaking Giskard which is trying to [import the file](https://github.com/Giskard-AI/giskard/blob/750db5c8f87193118d64d71ccbfe513e5132ecb7/giskard/rag/metrics/ragas_metrics.py#L17).

The help message is not informative and even misleading so it would be good to change it:
```
ImportError: Package ragas.llms.prompt is missing, it is required for the computation of RAGAS metrics. You can install it with `pip install ragas.llms.prompt`.
```


### Standalone code OR list down the steps to reproduce the issue

```shell
from giskard.rag.metrics.ragas_metrics import ragas_context_recall, ragas_context_precision
```


### Relevant log output

```shell
giskard\rag\metrics\ragas_metrics.py", line 17, in <module>
    from ragas.llms.prompt import PromptValue
ModuleNotFoundError: No module named 'ragas.llms.prompt'

giskard\rag\metrics\ragas_metrics.py", line 23, in <module>
    raise ImportError(
ImportError: Package ragas.llms.prompt is missing, it is required for the computation of RAGAS metrics. You can install it with `pip install ragas.llms.prompt`.
```


Hello @PierreMesure, we have just released giskard v2.15.3 which solves this bug!
Repository: Giskard-AI/giskard-oss
Issue: Outdated error message
Root cause: Documentation Deficiency

### Issue Type

Bug

### Source

source

### Giskard Library Version

latest

### OS Platform and Distribution

_No response_

### Python version

_No response_

### Installed python packages

_No response_

### Current Behaviour?

According to [this error message](https://github.com/Giskard-AI/giskard/blob/c0b167d4da1923c474ddf817234195c9c5ca031c/giskard/rag/metrics/ragas_metrics.py#L109), it's possible to pass a parameter called "retrieved_documents" but [it seems to be deprecated](https://github.com/orgs/Giskard-AI/discussions/1945#discussioncomment-9649292).

### Standalone code OR list down the steps to reproduce the issue

Run evaluate() with an answer_fn returning str instead of AgentAnswer.

### Relevant log output

_No response_

Hey @PierreMesure - Thanks for raising this. 

@henchaves would you have some time to have a look?

Hey @PierreMesure, thanks for reporting this.
A PR was opened to update the error message: https://github.com/Giskard-AI/giskard/pull/2062
It well be merged soon!
Repository: Giskard-AI/giskard-oss
Issue: Testset saved as ".json", not ".jsonl" when saving report
Root cause: Validation Gap

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.15.3

### OS Platform and Distribution

_No response_

### Python version

_No response_

### Installed python packages

_No response_

### Current Behaviour?

When saving a report, the testset.json file containing the testset contains JSONL so its file extension should reflect it.
I suggest changing the file name from `testset.json` to `testset.jsonl` [here](https://github.com/Giskard-AI/giskard/blob/d2ae5914828674c855b9a4181f48a984108deb2f/giskard/rag/report.py#L155). I can send a PR but not sure whether this will have unforeseen consequences.

### Standalone code OR list down the steps to reproduce the issue

Save a report

### Relevant log output

_No response_
Repository: Giskard-AI/giskard-oss
Issue: infinite build times on hf private spaces using docker image
Root cause: Unimplemented Feature Gap

### Issue Type

Bug

### Source

docker

### Giskard Library Version

not sure i just used the gui

### OS Platform and Distribution

huggingface spaces (not sure , used the image from hf)

### Python version

not sure , didnt check, sorry !

### Installed python packages

```shell
from docker image
```


### Current Behaviour?

```shell
infinite build times on private spaces deployment on huggingface
```


### Standalone code OR list down the steps to reproduce the issue

```shell
using the hf gui to launch docker image on private spaces
```


### Relevant log output

```shell
https://huggingface.co/spaces/open-acc/giskard?logs=build

===== Build Queued at 2024-11-20 19:52:18 / Commit SHA: ecc877b =====
```


Hey @Josephrp  - Sorry, we've deprecated this Docker image in favour of a newer product: https://www.giskard.ai/products/llm-evaluation-hub

We don't distribute this product on HF spaces but via hosted SaaS or on-private-Cloud k8s installation. Could you give us more information of you AI use case and testing need?

@Inokinoki could you add a note on the documentation on huggingface side to remove it and mark the deprecation?

well noted with thanks , i'll reach out to both huggingface and giskard with the information requested above
Repository: Giskard-AI/giskard-oss
Issue: num_samples and  num_requirements don't work in all detectors
Root cause: Documentation Deficiency

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.16.0

### OS Platform and Distribution

macos 14

### Python version

3.11

### Installed python packages

_No response_

### Current Behaviour?

```shell
The parameters num_samples=2, num_requirements=2 for example work only for some detectors like llm_harmful_content and the llm_stereotypes_detector.

The other detectors aren't affected by these parameters at all.
```


### Standalone code OR list down the steps to reproduce the issue

```shell
params = {
    "information_disclosure": dict(num_samples=2, num_requirements=2),
 
}
report = giskard.scan(model=giskard_model, only=["information_disclosure"], params=params)
```


### Relevant log output

```shell
🔎 Running scan…
Estimated calls to your model: ~20
Estimated LLM calls for evaluation: 25
```


Hello @AmineDjeghri,
Actually your `params` dict should contain the detector as the key and not the tag. For information disclosure, the detector name is `llm_information_discloure`, so the params should be:

```
params = {
    "llm_information_disclosure": dict(num_samples=2, num_requirements=2),
 
}
```

Could you test it again and tell me if you managed to reduce the number of calls to your model? The expected number is `num_samples * num_requirements`, so for that case should be 4 calls.
Repository: Giskard-AI/giskard-oss
Issue: giskard.rag.report.save fails if evaluate doesn't include KnowlegeBase
Root cause: Validation Gap

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.16.0

### OS Platform and Distribution

OL9

### Python version

3.11

### Installed python packages

_No response_

### Current Behaviour?

```shell
Running an evaluate without knowledge_base:

report = evaluate(get_answer, testset=loaded_testset)
```

An attempt to save the report:
```
report.save("/tmp/report")
```

Results in:
```
  File "/app/.venv/lib64/python3.11/site-packages/giskard/rag/report.py", line 161, in save
    self._knowledge_base._knowledge_base_df.to_json(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute '_knowledge_base_df'
```

Expected the following files to be written to /tmp:
* testset.jsonl
* report_details.json
* metrics_results.json

Maybe empty files of:
* knowledge_base.jsonl
* knowledge_base_meta.json

So that they can be load can be run on them later.
```


### Standalone code OR list down the steps to reproduce the issue

```shell
loaded_testset = QATestset.load("/tmp/ts/testset.jsonl")
report = evaluate(get_answer, testset=loaded_testset)
report.save("/tmp/report")
```


### Relevant log output

```shell
/app/.venv/lib64/python3.11/site-packages/giskard/rag/report.py", line 161, in save
    self._knowledge_base._knowledge_base_df.to_json(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute '_knowledge_base_df'
```


Hey @henchaves - Is it already logged?

Hello @gotsysdba,
Thanks for reporting this issue.

The bug was fixed and the updated methods will be available in the next release (or you can already access them on `main` branch). We plan to release a new giskard version still this week!
Repository: Giskard-AI/giskard-oss
Issue: KeyError: 'correctness' when running evaluate in giskard.rag
Root cause: Validation Gap

When running evaluate from giskard.rag with the specified metrics ragas_context_recall and ragas_context_precision, the following error occurs

```bash
from giskard.rag import evaluate, RAGReport, AgentAnswer
from giskard.rag.metrics.ragas_metrics import ragas_context_recall, ragas_context_precision

def answer_fn(question, history=None):

    answer = chat_engine.chat(question, chat_history=[])

    return AgentAnswer(
        message=answer.response,
        documents=[source.content for source in answer.sources]
    )

report = evaluate(answer_fn, 
                testset=testset, 
                knowledge_base=knowledge_base,
                metrics=[ragas_context_recall, ragas_context_precision])
report.save("test_report")
```
**ERROR**

```bash
KeyError                                  Traceback (most recent call last)
File c:\Users\KUNJAN SHAH\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\indexes\base.py:3805, in Index.get_loc(self, key)
   3804 try:
-> 3805     return self._engine.get_loc(casted_key)
   3806 except KeyError as err:

File index.pyx:167, in pandas._libs.index.IndexEngine.get_loc()

File index.pyx:196, in pandas._libs.index.IndexEngine.get_loc()

File pandas\\_libs\\hashtable_class_helper.pxi:7081, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas\\_libs\\hashtable_class_helper.pxi:7089, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'correctness'```

Hey @henchaves - is this a known issue?

Hello @kunjanshah0811, thanks for reporting this issue.

Could you confirm that you only get this error when calling `report.save` method? I've just tried to call this method and it's working fine here, so I think I'll need more information about your specs so I can try to simulate your env. Could you share what is you OS, Python version and pip list? 

Just so you know, we constrained the version of some external libs (such as `ragas` and `langchain`) which were causing incompatibilities with `generate_testset` and `evaluate` methods. You can try to use this `giskard` version while it's not released yet: https://github.com/Giskard-AI/giskard/pull/2122.





Something that could be happening: 

1. We always compute `correctness`, even if here only `ragas_context_recall `and `ragas_context_precision `are passed. So it explains why `correctness `is involved in the error. 

2. Now we are using a LLM to generate JSON, and here the error seems to be `KeyError: 'correctness'`, which may mean that the LLM that has been used just failed to generate the correctness key. 

So it may just be an error in the evaluate function and a LLM problem.
Repository: Giskard-AI/giskard-oss
Issue: 'gikard' is not a package
Root cause: Environment Incompatibility

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.16.1

### OS Platform and Distribution

Linux Ubuntu 24

### Python version

3.11

### Current Behaviour?

I literally tried to follow the eval test set building guide from [readme](https://github.com/Giskard-AI/giskard?tab=readme-ov-file#3--automatically-generate-an-evaluation-dataset-for-your-rag-applications). However, when I run the script I get error:

```shell
Traceback (most recent call last):
  File "/home/erykmazus/work/fennemore/./research/rag-eval-dataset-construction/giskard.py", line 5, in <module>
    from giskard.rag import generate_testset, KnowledgeBase
  File "/home/erykmazus/work/fennemore/research/rag-eval-dataset-construction/giskard.py", line 5, in <module>
    from giskard.rag import generate_testset, KnowledgeBase
ModuleNotFoundError: No module named 'giskard.rag'; 'giskard' is not a package
```

I installed it using:
```shell
pip install "giskard[llm]" -U
```

### Standalone code OR list down the steps to reproduce the issue

```shell
from giskard.rag import generate_testset, KnowledgeBase
```
Repository: Giskard-AI/giskard-oss
Issue: Update litellm version to latest
Root cause: External Dependency Breakage

Can you please update the litellm pacakge to the latest version ?
https://github.com/Giskard-AI/giskard/blob/a719f656d913bdebbac13e90d8b4bb0abe072e2f/pyproject.toml#L189

Hey @henchaves - Is it doable or would there by side effects from upgrading litellm to latest?

Hi @AmineDjeghri, I'll check the viability of upgrading LiteLLM version. 
What LiteLLM feature specifically from a latest that you want to use?

Hi, I will have a look 👍 

Awesome @GTimothee, let me know if you need any help.

Hello,

The PR #2139 was merged, so I'm closing this issue 🙂