zhiminy/EvalEng huggingface dataset:
Repository: Giskard-AI/giskard-oss
Issue: Update Python API references
GitHub labels: enhancement

## 🚀 Feature Request

The API references need to be improved

## 🔈 Motivation

The current API references are a bit messy

## 🛰 Alternatives

<!-- A clear and concise description of any alternative solutions or features you've considered. -->

## 📎 Additional context

<!-- Add any other context or screenshots about the feature request here. -->
Repository: Giskard-AI/giskard-oss
Issue: Improve RAGET implementation
GitHub labels: enhancement

## 🚀 Feature Request

The RAGET implementation holds some implementation decisions that seem to lead to user errors.


## 🔈 Motivation

The current API design leaves room for user errors.

## 🛰 Alternatives

NA.

## 📎 Additional context

NA.


@henchaves what do you think about the QoL updates for the RAGET implementation? I saw some comments and get a similar sentiment when getting this to work during quick testings. WDYT about the outlined lines of work?

Thanks @davidberenstein1957 for raising this topic!
Answered on Slack

Hello, I would like to suggest two other points of improvement:
- In the evaluate function we first compute all the answers from the RAG engine and then we evaluate on them, without a backup being created. Therefore if my test set contains 1000s of samples and the evaluation failed for whatever reason I lose everything and I am forced to recompute all the answers, which is a waste of time and money. Maybe we could split the evaluate function in two?
- Maybe we should not compute correctness by default. It seems a little bit odd to ask the user to select metrics to compute and compute another one behind the scenes without him knowing it. Also, it can raise weird issues like  https://github.com/Giskard-AI/giskard/issues/2111 where the user specified ragas metrics and there is a failure related to correctness

Thanks for the help and I agree with both of these! I will create internal issues for them.



@GTimothee I created some subissues for those, feel free to respond and pick up any of the last two :)

Hi again, new pain points I identified: 
1. It seems like only correctness is supported by the report, so we will need to update the Report object. Indeed, I created a new metric `correctness_score` but the method `report.component_scores` explicitly (hard coded) filters metrics to only use and show the `correctness` metric in the report. So I defined and computed my custom metric, but in the end another metric is displayed in the report. 
2. Also, we need to implement a fix to handle the case where we return a text metric. I tried to return a `feedback` (string) along with my `correctness_score` but it makes the Report crash when you try to print or save it. This is because internally it calls histogram computations on all metrics (although it does not show them, refer to 1.), no matter their types. So I had to remove my feedback output.
Repository: Giskard-AI/giskard-oss
Issue: Add support and documentation for creating custom metrics in RAGET
Root cause: Documentation Deficiency

## 🚀 Feature Request

We rely on metrics from ragas but we don't clarify how to create our own metrics for RAGET. 

We can check how we can easily create a new metric on top of `Metric(ABC)` within `giskard.rag.metrics.base`. Similarly, we can take a look at `CorrectnessMetric` as a base example. As well as explore and describe how this can be done by anyone in the community.

## 🔈 Motivation

People want to design their own metrics.

## 🛰 Alternatives

<!-- A clear and concise description of any alternative solutions or features you've considered. -->

## 📎 Additional context

<!-- Add any other context or screenshots about the feature request here. -->


Taking this one :)
Repository: Giskard-AI/giskard-oss
Issue: KnowledgeBase requires recomputation of embeddings upon each run, we could have serialisation methods, or a method to load it with embeddings.
Root cause: Unimplemented Feature Gap


Will do!

Lovely. Thanks @GTimothee. I think the issues holds 2 sub-issues. Potentially, allowing to re-use, start with pre-computed embeddings, and adding serialisation to the KnowledgeBase (saving and loading locally, and to the Hub). We can copy the logic and behavior as much as possible from the TestDataset you worked on :) 

fyi working on it ;) (https://huggingface.co/datasets/GTimothee/my-knowledge-base)
Repository: Giskard-AI/giskard-oss
Issue: Allow users to define their own metrics and don't force using correctness
Root cause: Architectural Constraint

w.r.t. the correctness, I think internally there are some niche methods that would need to be overhauled too.

like report.correctness_by_topic()

 instead we could do something like metrics_by_topic("correctness") or something

So the idea would be to remove the computation of correctness by default, and therefore change the report so that it only displays what it can, right ? Because so far I think the report always try to show correctness by components. 

> Because

Yes. I think we can see if we could open this up easily; if not, we can leave it for a more serious refactor and perhaps focus on another issue. I have not looked at the current setup. Still, ideally, we want it to be backwards compatible and don't want to make major/breaking changes, which might be especially difficult in the report generation.
Repository: Giskard-AI/giskard-oss
Issue: split evaluate and answer generation
GitHub labels: enhancement

## 🚀 Feature Request

<!-- A clear and concise description of the feature proposal. -->

## 🔈 Motivation

<!-- Please describe the motivation for this proposal. -->

## 🛰 Alternatives

<!-- A clear and concise description of any alternative solutions or features you've considered. -->

## 📎 Additional context

<!-- Add any other context or screenshots about the feature request here. -->


Is this open to work?


Hi @staru09, This is on hold for the moment, but there are some [Good First Issues your could work on](https://github.com/Giskard-AI/giskard/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22). 
Besides that, there are also some chores we have been postponing that would help out the community a lot:
- https://github.com/Giskard-AI/giskard/issues/2154
- https://github.com/Giskard-AI/giskard/issues/2173

Cool. I'll get started with these.
Repository: Giskard-AI/giskard-oss
Issue: [chore] bump version `faiss-cpu`
Root cause: Architectural Constraint

## 🚀 Feature Request

Currently the version of `faiss` is capped at 1.8, but it can be `<1.11`.

## 🔈 Motivation
Wider support

## 🛰 Alternatives

NA
## 📎 Additional context

NA
Repository: Giskard-AI/giskard-oss
Issue: [bug] ModuleNotFoundError: No module named 'IPython'
Root cause: Validation Gap

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.16.2

### OS Platform and Distribution

_No response_

### Python version

_No response_

### Installed python packages

```shell

```

### Current Behaviour?

```shell
giskard/rag/report.py 

from IPython.core.display import HTML
ModuleNotFoundError: No module named 'IPython'
```

### Standalone code OR list down the steps to reproduce the issue

```shell
run a report without havving ipython installed
```

### Relevant log output

```shell

```
Repository: Giskard-AI/giskard-oss
Issue: [chore] bump Python version to `3.13`
Root cause: Unimplemented Feature Gap

## 🚀 Feature Request

3.13 could be supported

## 🔈 Motivation

It has been around for a while.

## 🛰 Alternatives

<!-- A clear and concise description of any alternative solutions or features you've considered. -->

## 📎 Additional context

<!-- Add any other context or screenshots about the feature request here. -->
Repository: Giskard-AI/giskard-oss
Issue: Can I use repo for evaluating on-premise offline LLM powered with RAG without OpenAI ?
Root cause: Documentation Deficiency

## Checklist

<!-- Mark with an `x` all the checkboxes that apply (like `[x]`) -->

- [ ] I've searched the project's [`issues`](https://github.com/Giskard-AI/giskard/issues?q=is%3Aissue).

## ❓ Question

We are using LLama LLM with RAG, can i use the repo for evaluating generated answers without using OpenAI as our company does not allow connecting to online APIs due to confidentiality of our data. 


## 📎 Additional context

<!-- Add any other context or screenshots about the feature request here. -->


Yes you can. You can, I recommend taking a look at our Rage Evaluation Toolkit https://docs.giskard.ai/en/stable/open_source/testset_generation/index.html
Repository: Giskard-AI/giskard-oss
Issue: bump numpy version to >2
Root cause: External Dependency Breakage

## 🚀 Feature Request
- bump numpy version to >2

## 🔈 Motivation
- there are conflicts with other libraries that have numpy>2
- numpy v2 has been out for a year - the ecosystem moved toward it


Thank you for the amazing project!

Hi @fazpu would you be able to open a PR for this? It should be straightforward enough :)
Repository: Giskard-AI/giskard-oss
Issue: Faster import
Root cause: Architectural Constraint

## 🚀 Feature Request

Importing giskard takes almost a minute

## 🔈 Motivation
Too long importing time
<img width="269" alt="Image" src="https://github.com/user-attachments/assets/c92e5e66-910d-4362-8e99-14ad17a18911" />


Hi @khoaguin because our libary is rather big and support many dependencies, we recommend importing smaller segment to speed this up. something like `from giskard.rag import generate_testset, KnowledgeBase` should already be much faster. 

Thanks for raising the issue but for now we will not spend time on refactoring the library with the idea of speeding up priorities.

Thanks for the info @davidberenstein1957
Repository: Giskard-AI/giskard-oss
Issue: Segmentation Fault
Root cause: Environment Incompatibility

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.17.0

### OS Platform and Distribution

mac os 

### Python version

3.12

### Installed python packages

```shell
Followed the github url for the starter guide - works on google collab
on local env it crashes

i user p.ython 3.12
and 3.10
```

### Current Behaviour?

```shell
tokenizer_config.json: 100%|█████████████████████████████████████| 49.0/49.0 [00:00<00:00, 187kB/s]
config.json: 100%|████████████████████████████████████████████████| 466/466 [00:00<00:00, 1.96MB/s]
vocab.txt: 100%|████████████████████████████████████████████████| 996k/996k [00:00<00:00, 4.16MB/s]
tokenizer.json: 100%|█████████████████████████████████████████| 1.96M/1.96M [00:00<00:00, 2.31MB/s]
model.safetensors: 100%|████████████████████████████████████████| 542M/542M [02:45<00:00, 3.27MB/s]
zsh: segmentation fault  python climate_agent.py
/Users/neelkanani/.pyenv/versions/3.10.13/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaph
```

### Standalone code OR list down the steps to reproduce the issue

```shell
I try to run and it fails
```

### Relevant log output

```shell

```

Hi @neelkanani, it's cool that you are testing out our package. Could you share some more info with me on this? What code are you running?
Repository: Giskard-AI/giskard-oss
Issue: Use of deprecated SciPy API
Root cause: External Dependency Breakage

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.18.0

### OS Platform and Distribution

WSL

### Python version

3.12.4

### Installed python packages

```shell

```

### Current Behaviour?

```shell
When attempting to import giskard, the following error occurs:
`ImportError: cannot import name 'Ks_2sampResult' from 'scipy.stats.stats'`

(This happened with `from giskard import Model, Dataset, scan, testing, demo`, but persisted when running it with each of the items separately, as well).

It appears the scipy class was removed/renamed in the newer versions (according to GPT5, in recent versions, ks_2samp returns KstestResult, and Ks_2sampResult is no longer exported).
```

### Standalone code OR list down the steps to reproduce the issue

```shell
`from giskard import Model, Dataset, scan, testing, demo`
```

### Relevant log output

```shell

```
Repository: Giskard-AI/giskard-oss
Issue: Add a Pull Request (PR) Template to standardize contributions


**Description:**
Potential Benefits: Consistent PRs, easier reviews, clear checklists.
Suggested to .github/PULL_REQUEST_TEMPLATE.md for a standard template.
Attaching a sample template (see below):

# Pull Request Template

Thank you for contributing to Giskard! Please fill out this template to help us review your changes efficiently.

## Description

<!-- Briefly describe your changes and the motivation behind them.
   If this PR addresses an issue, please link it (e.g., closes #123). -->

## Type of Change

- [ ] Bug fix
- [ ] Documentation update
- [ ] Example/tutorial improvement
- [ ] New ML test (performance, drift, metamorphic, etc.)
- [ ] New feature
- [ ] Refactor
- [ ] Other (please describe):

## Checklist

- [ ] My code follows the [style guide](https://docs.giskard.ai/en/stable/community/contribution_guidelines/dev-environment.html) (`black`, `isort`, pre-commit hooks).
- [ ] I have added or updated relevant docstrings in [numpydoc format](https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard).
- [ ] I have updated the documentation/examples as needed.
- [ ] I have added tests or updated existing tests if needed.
- [ ] I have verified that my code works with the supported Python versions.
- [ ] I have performed a self-review of my code.
- [ ] I have checked that the PR is not duplicating an existing issue/PR.

## Additional Information

*Add any extra context, screenshots, or deployment notes for reviewers.*

## How to Reproduce/Test

*Explain how reviewers can test your changes. Add code snippets or steps if helpful.*
Repository: Giskard-AI/giskard-oss
Issue: Is it normal that the run method of Detectors doesn't use the given dataset for evaluation?
Root cause: Validation Gap

## Checklist

- [X] I've searched the project's [`issues`](https://github.com/Giskard-AI/giskard/issues?q=is%3Aissue).

## ❓ Question

I want to use a local dataset with the `LLMInformationDisclosureDetector`, but it doesn't seem that the evaluator uses it. Is it intended? If not, how can I use such detector without generating test cases? I want to only use the one in my file.

## 📎 Additional context

I think it's a follow-up to the Issue #1893 . Using the given answer, the detector still doesn't use the given dataset for evaluation, it still generate test cases.
Moreover, in giskard.scanner.llm.base.py script the given dataset is only used for language detection.  https://github.com/Giskard-AI/giskard-oss/blob/752a81fa9f083947ab930cfc6929ca02ff43463b/giskard/scanner/llm/base.py#L72
Repository: Giskard-AI/giskard-oss
Issue: Giskard Installation Issues Due to Incompatible Package Versions
Root cause: External Dependency Breakage

### Issue Type

Build/Install

### Source

source

### Giskard Library Version

2.18.0

### OS Platform and Distribution

_No response_

### Python version

3.12

### Installed python packages

```shell

```

### Current Behaviour?

```shell
I encountered challenges while installing Giskard due to version incompatibilities between Python and its dependencies. Specifically:

Dependency mismatches: Some required packages have minimum or maximum version constraints that do not align with the versions supported by Python 3.12 (e.g., SciPy, scikit-learn).

Unavailable “middle” versions: For certain dependencies, the version ranges listed by Giskard skip intermediate versions that are otherwise compatible, making it difficult to satisfy all requirements simultaneously.

Installation hurdles: As a result, users are forced to either downgrade Python or use older/newer versions of dependencies that may not be fully compatible with Giskard, complicating the setup process.

Suggestion:
It would be helpful if Giskard updated its dependency requirements to fully support modern Python versions (3.12+), or provided a clear compatibility matrix showing which versions of dependencies work together. This would simplify installation and reduce friction for users.
```

### Standalone code OR list down the steps to reproduce the issue

```shell
Dependency mismatches: Some required packages have minimum or maximum version constraints that do not align with the versions supported by Python 3.12 (e.g., SciPy, scikit-learn).

Unavailable “middle” versions: For certain dependencies, the version ranges listed by Giskard skip intermediate versions that are otherwise compatible, making it difficult to satisfy all requirements simultaneously.

Installation hurdles: As a result, users are forced to either downgrade Python or use older/newer versions of dependencies that may not be fully compatible with Giskard, complicating the setup process.
```

### Relevant log output

```shell

```
Repository: Giskard-AI/giskard-oss
Issue: No 'good first issues'.
GitHub labels: question

## No Good First Issues.

<!-- What is your question -->

As a budding community, it's necessary to attract the new talent towards one's organisation. And 'G.F.I.' is the best practice to do that. But I didn't find any such issues. 

## 📎 Additional context

<!-- Add any other context or screenshots about the feature request here. -->
A few good first issues can be:
- Modifying the documentations.
- Updating tags such as version, Licence etc in the readme.md and other documents whenever there's a new update. 
- Adding hyperlinks to the necessary models (such as PyTorch, TensorFlow etc) to make the docs more interactive.
- Adding operating demos about a particular feature.
- Other issues that are time consuming but simple and easy.
As long as there's something for the newcomers to do... The community will grow.


Now that we've migrated your tasks to GitHub, we'll be adding [d:easy](https://github.com/Giskard-AI/giskard/labels/d%3Aeasy) tag to mark easy tasks that could be good starting points for anyone who wants to contribute to Giskard
Repository: Giskard-AI/giskard-oss
Issue: File not found
Root cause: Documentation Deficiency

The PFA file *Worker* is not found. And the said folder contains various errors when opened in my IDE.


https://github.com/Giskard-AI/giskard/blob/92d1deb14bbe47a7df222def72d5297ce0cbb50f/giskard-server/src/main/java/ai/giskard/ml/MLWorkerClient.java#L10

![Screenshot from 2022-09-27 23-23-11](https://user-images.githubusercontent.com/60355338/192600796-74bd5fb1-6a49-43af-9f7a-7d89ed19e2d1.png)



I guess the said file is either removed or moved to a new destination. If possible can someone guide me on how to resolve this issue? I'm just starting with open-source so please bear with me.😅

Hi @p4puniya ,
The missing files are auto-generated during the build of the project. To run the build you can execute `./gradlew build` from the root of the cloned repo.

We're working on improving developers' documentation, for now, there's a Notion page that describes how to set up Giskard for local development using Intellij Idea (but you may find it useful for VS Code too).
https://giskard.notion.site/Running-Giskard-on-local-without-docker-6b5b4f47a0da4b4a9851bbf65199c5cd


Hi @andreybavt ,
I tried building the project but it shows *APPLICATION FAILED TO START*
Here's the description and the action:
Description:
Parameter 2 of constructor in ai.giskard.service.ProjectService required a bean of type 'ai.giskard.web.dto.mapper.GiskardMapper' that could not be found.
Action:
Consider defining a bean of type 'ai.giskard.web.dto.mapper.GiskardMapper' in your configuration.

Hello @p4puniya, 

Reposting this here from discussion on discord (thanks for the help).

This should solve your problem:

> classes from ai.giskard.worker are autogenerated from GRPC proto file located in giskard-server/ml-worker-proto/proto/ml-worker.proto
> 
> This ml-worker-proto directory is a submodule, so to fetch it you can run this command from the root of giskard repository
> `git submodule update --init --recursive -q`
> 
> next, to generate the missing java files you can run
> `./gradlew :giskard-server:cleanGenerated :giskard-server:generateProto`

Hi everyone,

We decided to simplify Giskard repo structure and merged 3 repos (giskard, giskard-client, and ml-worker-proto) into the main `giskard` repo. This allowed us to get rid of the proto file submodule, so no need to pull submodule recursively now. 

Starting giskard in dev mode should be now done with just

`./gradlew start --parallel`

Check out the dev readme for more details
https://github.com/Giskard-AI/giskard/blob/main/README.dev.md
Repository: Giskard-AI/giskard-oss
Issue: H2O models unable to unpickle on Giskard 
Root cause: External Dependency Breakage

## 🐛 Bug Report

The bug occurs during `_validate_model_is_pickleable` of `H2OAutoML`:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/Documents/giskard-client/.venv/lib/python3.7/site-packages/giskard/client/project.py in _validate_model_is_pickleable(prediction_function)
    674             pickled_model = cloudpickle.dumps(prediction_function)
--> 675             unpickled_model = cloudpickle.loads(pickled_model)
    676         except Exception:

TypeError: __new__() missing 1 required positional argument: 'keyvals'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-55-c024ce4c5c6b> in <module>
     34     classification_labels=['Yes','No'],
     35     model_name='test_model',
---> 36     dataset_name='test_data'
     37     )

~/Documents/giskard-client/.venv/lib/python3.7/site-packages/giskard/client/project.py in upload_model_and_df(self, prediction_function, model_type, df, column_types, feature_names, target, model_name, dataset_name, classification_threshold, classification_labels)
    396             prediction_function,
    397             target,
--> 398             df,
    399         )
    400         data_res = self._post_data(column_types, data, dataset_name, raw_column_types, target)

~/Documents/giskard-client/.venv/lib/python3.7/site-packages/giskard/client/project.py in _validate_model(self, classification_labels, classification_threshold, feature_names, model_type, prediction_function, target, validate_df)
    222             validate_df,
    223     ):
--> 224         prediction_function = self._validate_model_is_pickleable(prediction_function)
    225         transformed_pred_func = self.transform_prediction_function(
    226             prediction_function, feature_names

~/Documents/giskard-client/.venv/lib/python3.7/site-packages/giskard/client/project.py in _validate_model_is_pickleable(prediction_function)
    675             unpickled_model = cloudpickle.loads(pickled_model)
    676         except Exception:
--> 677             raise ValueError("Unable to pickle or unpickle model on Giskard")
    678         return unpickled_model
    679 

ValueError: Unable to pickle or unpickle model on Giskard
```

## 🔬 How To Reproduce

Run the following script: -- (TBD)

### Environment

- Environment independent problem, but tried on: Windows / macOS.  
- Python version independent problem, but tried with both 3.7 and 3.10

## 📎 Additional context
Similar mentions by other people too (like https://groups.google.com/g/h2ostream/c/FaJm57Rbkmg?pli=1)
One of the proposed solution (to be tested) is to:
- use the internal save_model function in H2O,
- load it in to python as a binary object,
- pickle the binary instead.


H2O models are currently not supported.
Repository: Giskard-AI/giskard-oss
Issue: Tensorflow model integration with Adam optimizer: IMDB notebook
Root cause: Unimplemented Feature Gap

[GSK-273](https://giskard.youtrack.cloud/issue/GSK-273)



@aavtomonov: to verify this card I would need that gsk supports : tensorrflow, tensorflow text



Project: Giskard
Reporter: jmjm
Created: 2022-09-09