zhiminy/EvalEng huggingface dataset:
Repository: Giskard-AI/giskard-oss
Issue: Dataset  feature on the doc is empty

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.0.0b

### Giskard Server Version

1.3.0

### OS Platform and Distribution

Windows

### Python version

3.9

### Installed python packages

_No response_

### Current Behaviour?

```shell
The dataset feature for wrapping a dataset with giskard is empty.
```


### Standalone code OR list down the steps to reproduce the issue

```shell
https://docs.giskard.ai/en/latest/reference/datasets/index.html#dataset

Check this out and you see it's empty
```


### Relevant log output

_No response_

It's been resolved.
The link works now
Repository: Giskard-AI/giskard-oss
Issue: Is it possible to use Giskard for computer vision / images?
Root cause: Unimplemented Feature Gap

## Checklist

- [x] I've searched the project's [`issues`](https://github.com/Giskard-AI/giskard/issues?q=is%3Aissue).

## ❓ Question

<!-- What is your question -->

As in title, is it possible to use Giskard for computer vision, i.e. image datases? I am using HuggingFace for vision CNN models (ResNet, ConvNeXt etc.) and would like to test them.


Thanks for the question @j-adamczyk ! We're in the process of working on support computer vision. Would be great to get your feedback on our upcoming work.

@luca-martial Any chance you are starting to work on this for computer vision? I would be happy to research the state-of-the art about perturbation methods for binary classification for CV let's say to get started. Let me know if this could help. :ok_hand: 

> @luca-martial Any chance you are starting to work on this for computer vision? I would be happy to research the state-of-the art about perturbation methods for binary classification for CV let's say to get started. Let me know if this could help. 👌

@yassineAlouini Thanks for suggesting this! We'll get back to you asap with more info!

> @luca-martial Any chance you are starting to work on this for computer vision? I would be happy to research the state-of-the art about perturbation methods for binary classification for CV let's say to get started. Let me know if this could help. 👌

Hey @yassineAlouini, now we have a dedicated repo for CV: https://github.com/Giskard-AI/giskard-vision

> > @luca-martial Any chance you are starting to work on this for computer vision? I would be happy to research the state-of-the art about perturbation methods for binary classification for CV let's say to get started. Let me know if this could help. 👌
> 
> Hey @yassineAlouini, now we have a dedicated repo for CV: https://github.com/Giskard-AI/giskard-vision

That's great news!
Repository: Giskard-AI/giskard-oss
Issue: Update dependencies to Pandas 2.X
Root cause: External Dependency Breakage

## 🚀 Feature Request

Currently Giskard depends on Pandas 2.X, at least Giskard 2.0.0. Keeping this up to date would make it easier to manage dependencies.


@j-adamczyk thanks for the feedback, we're working on this internally, will let you know once work is done

Hey @luca-martial  - Is this issue still open or should we close it?
Repository: Giskard-AI/giskard-oss
Issue: Feature: add OCR typos transformation to transformation functions catalog
Root cause: Unimplemented Feature Gap

## 🚀 Feature Request

<!-- A clear and concise description of the feature proposal. -->
Giskard currently implements a [regular typo transformation](https://github.com/Giskard-AI/giskard/blob/main/giskard/scanner/robustness/text_transformations.py#L70) in its robustness scan. We are looking to implement a similar transformation which is based on typos that frequently occur after OCR processing.

Example: "Their cat eats everyday" -> "Tbeir c^at eats evervd^av"

## 🔈 Motivation

<!-- Please describe the motivation for this proposal. -->
This will enhance the completeness of Giskard's catalog.


I believe I can work on this well, can you kindly assign this to me? @luca-martial 

Thanks @goyalpramod ! I'm assigning you now
Repository: Giskard-AI/giskard-oss
Issue: Feature: add speech-to-text typos transformation to transformation functions catalog
Root cause: Unimplemented Feature Gap

## 🚀 Feature Request

<!-- A clear and concise description of the feature proposal. -->
Giskard currently implements a [regular typo transformation](https://github.com/Giskard-AI/giskard/blob/main/giskard/scanner/robustness/text_transformations.py#L70) in its robustness scan. We are looking to implement a similar transformation which is based on typos that frequently occur after speech-to-text processing.

Example: "NASDAQ dropped by 2.1%" -> "Attack drop by 21%"

## 🔈 Motivation

<!-- Please describe the motivation for this proposal. -->
This will enhance the completeness of Giskard's catalog.


Hello @luca-martial I would like to work on this issue, could you kindly assign it to me? 
I had a few initial questions as well. 
1. I believe the structure will be similar to TextTypoTransformation but we have to take into consideration that mistakes that are made in speech to text for example "Too" may be misclassified as "Two". So the code should reflect that. (Assuming English is the primary target language) 
2. How are filler words supposed to be handled? for example "humm", "uh", "ah". 

Thanks @goyalpramod ! For (1) yes exactly, for (2) I don't believe you will need to transform filler words. Let me know if you have any other question!
Repository: Giskard-AI/giskard-oss
Issue: Feature: add number-to-word transformation to transformation functions catalog
Root cause: Unimplemented Feature Gap

## 🚀 Feature Request

<!-- A clear and concise description of the feature proposal. -->
Giskard currently implements a [transformations](https://github.com/Giskard-AI/giskard/blob/main/giskard/scanner/robustness/text_transformations.py) in its robustness scan. We are looking to implement a new transformation which transforms numbers to words.

Example: "Around 153 people were recovered" -> "Around one hundred fifty three people were recovered"

## 🔈 Motivation

<!-- Please describe the motivation for this proposal. -->
This will enhance the completeness of Giskard's catalog.


Hello Luca, Is it fair to assume the text would be only in the English language? Do we want to handle the target language as a choice of input?

There is a Python package [num2words](https://github.com/savoirfairelinux/num2words) that converts numbers to words and has multi-language support.

@sagar118 thanks for your interest! It's fair to say that text will be in English for > 80% of cases. Multi-language support would be a nice addition though. I'll let you decide based on feasibility!

Sounds good.

Do we want to convert all numbers to text? For instance, numbers can be part of dates, emails, and URLs. Hence, should the expected behavior be to change all numbers to text or only for a particular scenario like plain numbers?

@sagar118 you can start with a v1 that converts all numbers. We can reassess later if needed!

Sure. That sounds good.
Repository: Giskard-AI/giskard-oss
Issue: can this giskard be used for codellama?
Root cause: Documentation Deficiency

Hi 
team,

Does it make sense to use giskard where the input is a text and output is a code?

Hello @pearlmary 

You can indeed use Giskard with LLMs for code generation. For example by using the LLM-as-a-judge test you can add a new test asking it to create a function and have the requirement check that the function appears to be correct.

This however leaves a caveat: It relies on how good the LLM is at evaluating code.

While there currently aren't tests in our catalog for code evaluation, it should be possible to create a new test that will:

1. Generate text based on your prompt
2. Receive code as output
3. Start evaluating the code

And define new inputs on that test, taking in your function arguments.

Thank you for the clarifications,
One more basic question, 
Can we replace openAI with any other open source LLM-as-a-judge. Do you have any examples or the syntax followed if it is possible to use other LLMs

You can use any LLM. Giskard define a "standard" library of tests, but all of them are open source. For example, the LLM-as-a-judge tests can be found here: https://github.com/Giskard-AI/giskard/blob/main/giskard/testing/tests/llm/output_requirements.py#L77-L131 

From this link, you'll see a basic example. You can then write a similar function in your own environment and use the decorator so that it can be plugged into the Giskard python library or the Giskard Hub seamlessly !

Other references: 
https://github.com/Giskard-AI/giskard/blob/main/giskard/llm/evaluators/base.py

Thank you for the quick reply. 
Trying to understand your point though.
LLM scan requires an OpenAI API key. Can I replace it with any other llms such as llama 2. 
Have you any example tutorial notebooks, that has used an open source LLMs for performing LLM scan? Any provision to do it a right away?

Thanks in Advance.

Hello @pearlmary 

The scan currently use different evaluation technics and some are coupled with using OpenAI (specifically GPT-4).

However your model can use llama2, the workflow is the following:

1. We generate text using your model (llama2, GPT, Cohere, ...)
2. We evaluate the output of your model using GPT-4

You can find detailed information of our evaluator in [the documentation](https://docs.giskard.ai/en/latest/knowledge/llm_vulnerabilities/index.html), you'll see that we have LLM assisted evaluators that only works with GPT-4.

We are aware of the issues (require API key, send info to Open AI, ...) and are working on custom evaluators that aren't coupled with GPT-4.
Repository: Giskard-AI/giskard-oss
Issue: [GSK-2179] Slicing function DB insert fails because of the long name
Root cause: Validation Gap

### Issue Type

Bug

### Source

docker

### Giskard Library Version

2.0.2

### Giskard Hub Version

2.0.2

### OS Platform and Distribution

macOS Ventura 13.4

### Python version

3.10

### Installed python packages

_No response_

### Current Behaviour?

```shell
I'm using featuretools lib to generate features. It can generate pretty long names for the features, like `TIME_SINCE_LAST_MIN(my_date_column123, my_numeric_column123)`. As a result, the automatic slicing function name is longer than 255 characters and a DB insert fails. I believe it makes sense to increase the limit.
```


### Standalone code OR list down the steps to reproduce the issue

```shell
results = scan(giskard_model, giskard_dataset, only=["performance"])
test_suite = results.generate_test_suite("performance")
test_suite.upload(client, project_name)
```


### Relevant log output

```shell
GiskardError: Internal Server Error: could not execute batch [Batch entry 0 insert into callable_functions (code,display_name,doc,module,module_doc,name,tags,version,cell_level,clauses,column_type,process_type,callable_type,uuid) values ('','`TIME_SINCE_LAST_MIN(my_date_column123, my_numeric_column123)` >= 14.512 AND `TIME_SINCE_LAST_MIN(my_date_column123, my_numeric_column123)` < 40748.859','Automatically generated slicing function',NULL,NULL,'`TIME_SINCE_LAST_MIN(my_date_column123, my_numeric_column123)` >= 14.512 AND `TIME_SINCE_LAST_MIN(my_date_column123, my_numeric_column123)` < 40748.859','["pickle","scan"]',1,'FALSE','[{"columnName":"TIME_SINCE_LAST_MIN(my_date_column123, my_numeric_column123)","comparisonType":"GREATER_THAN_EQUALS","value":14.512499809265137},{"columnName":"TIME_SINCE_LAST_MIN(my_date_column123, my_numeric_column123)","comparisonType":"LOWER_THAN","value":40748.859375}]',NULL,'CLAUSES','SLICING','af55a8b9-518f-5b71-bd1d-c16a649dec36'::uuid) was aborted: ERROR: value too long for type character varying(255)  Call getNextException to see other errors in the batch.] [insert into callable_functions (code,display_name,doc,module,module_doc,name,tags,version,cell_level,clauses,column_type,process_type,callable_type,uuid) values (?,?,?,?,?,?,?,?,?,?,?,?,'SLICING',?)]; SQL [insert into callable_functions (code,display_name,doc,module,module_doc,name,tags,version,cell_level,clauses,column_type,process_type,callable_type,uuid) values (?,?,?,?,?,?,?,?,?,?,?,?,'SLICING',?)]
```


@martjushev 

Thanks for reporting the issue, we will fix this issue ASAP
Repository: Giskard-AI/giskard-oss
Issue: Broken Documentation link in Readme
GitHub labels: documentation

good first issue

### Issue Type

Documentation Bug

### Source

source

### Giskard Library Version

2.0.3

### Giskard Hub Version

2.0.3

### OS Platform and Distribution

_No response_

### Python version

_No response_

### Installed python packages

_No response_

### Current Behaviour?

```shell
The documentation link on the Readme is broken: `https://docs.giskard.ai/en/latest/getting-started/quickstart.html` (404 Not Found) and should be replaced to `https://docs.giskard.ai/en/latest/index.html`
```


### Standalone code OR list down the steps to reproduce the issue

```shell
1. Just click on the main Documentation link in the readme: https://github.com/Giskard-AI/giskard
2. There's no step 2 :P
```


### Relevant log output

_No response_

Hi @simonamaggio thanks for reporting this! Were you planning on applying the fix as your first contribution? Otherwise we will take it internally!

I would like to work on this issue
Can you please assign this to me ?


Thanks @akhil888binoy !
Repository: Giskard-AI/giskard-oss
Issue: Update password hashing to use SHA-256 for enhanced security!
GitHub labels: enhancement

security

The change involves replacing the existing code `hashlib.md5(string.encode()).hexdigest()` with the more robust `hashlib.sha256(string.encode()).hexdigest()`.

This adjustment enhances resistance against potential security threats and aligns with best practices for cryptographic hash functions.

Hi, I would like to work on this issue, could you assign it to me?

Hello @PAARTH2608 

Unfortunately this issue require us to update the Giskard Hub that is closed source. Only updating the hash method for the Giskard Library client will produce authentication error (since another hash is expected).
Repository: Giskard-AI/giskard-oss
Issue: Reduce percentage float precision in display results details
Root cause: Unimplemented Feature Gap

## 🚀 Feature Request

Reduce percentage float precision in display results details

## 🔈 Motivation

The % might be shown with many useless decimals

## 🛰 Alternatives

Round up all % to 2 decimals

## 📎 Additional context

Example with Titanic:
For records in the dataset where `text_length(Name)` < 28.500, we found a significantly higher number of overconfident wrong predictions (33 samples, corresponding to 55.932203389830505% of the wrong predictions in the data slice). 

Great request @fran6w ! I'll mark this as a good first issue and see if community contributors are open to take it. 

Let me know if you prefer that I assign it to you instead!

Thank you, but I am not enough familiar with Giskard's code.

> Thank you, but I am not enough familiar with Giskard's code.

No problem, thanks for contributing a request!

I can help with this issue. I see you're already using 1 decimal in the `UnderconfidenceDetector`. Do you want to standardize it to 1 or 2 decimals? Please let me know and I'll update both accordingly.

> I can help with this issue. I see you're already using 1 decimal in the `UnderconfidenceDetector`. Do you want to standardize it to 1 or 2 decimals? Please let me know and I'll update both accordingly.

Great, 2 decimals is good. Note this should only be for the final display

I made the change in #1754, here's what it looks like now:

| Vulnerability | Description |
|---------------|-------------|
| Overconfidence | For records in the dataset where `text_length(sentence)` >= 65.500, we found a significantly higher number of overconfident wrong predictions (59 samples, corresponding to 22.01% of the wrong predictions in the data slice). |

PR #1754 has been merged. This issue may be closed.
Repository: Giskard-AI/giskard-oss
Issue: Add text transform to remove accents
Root cause: Unimplemented Feature Gap

## 🚀 Feature Request

Add a text transform to remove accents.

Example: `C'est l'été` becomes `C'est l'ete`

The transformation should be able to work with various characters and both lowercase and uppercase:

`çà et là` becomes `ca et la`
`Tiếng Việt` becomes `Tieng Viet`
`État` becomes `Etat`
and so on

It might be worth to have a perturbation rate parameters (default 100%)  

## 🔈 Motivation

We want to be able to test model with common text variation in languages using accents.



<p><a href="https://linear.app/giskard/issue/GSK-1069/add-text-transform-remove-accents">GSK-1069 Add text transform: remove accents</a></p>

Hi, I am interested in working on this issue.

> Hi, I am interested in working on this issue.

Great to hear that @asselindebeauville - I just assigned it to you!
Repository: Giskard-AI/giskard-oss
Issue: Refactor the `TestSuiteResult` structure
Root cause: Architectural Constraint

## 🚀 Feature Request

Currently the `TestSuiteResult` class uses a `List[Tuple[str, TestResult, Dict[str, Any]]])` for the `TestResult` information.

We need to have a `List[SuiteTestResult]` (`SuiteTestResult` does not exist yet) in order to greatly improve the code structure.

## 🔈 Motivation

This data structure is not humain readable and makes it hard to maintain (ei. We want to add an execution time for each test [in this PR](https://github.com/Giskard-AI/giskard/pull/1703))


<p><a href="https://linear.app/giskard/issue/GSK-2671/refactor-the-testsuiteresult-structure">GSK-2671 Refactor the `TestSuiteResult` structure</a></p>
Repository: Giskard-AI/giskard-oss
Issue: Scan: Overlapping slices should be grouped in the scan generated issues
Root cause: Unimplemented Feature Gap

## 🚀 Feature Request

If a slice is completely contained into another slice, we should just report the biggest one.

## 🔈 Motivation

It will makes the scan report more concise and avoid duplication. Furthermore it takes time and memory to check for those sub slices and it doesn't really provide any value.


Hi @kevinmessiaen ,

I noticed this issue and would like to contribute to it. Is it still open and relevant? If so, I would appreciate any guidance or additional information that could help me get started.

Thank you!

Hello @abhibongale 

Yes the issue is still relevant, we would appreciate your contribution on this one!

Basically in the `Scanner` (`giskard.scanner.scanner.py`) we run a bunch of evaluators depending of the model type.

For the `regression` and `classification` models, the detectors will be using the `SliceFinder` (`giskard.slicing.slice_finder.py`) to generate some slices that will then be tested. Some of those slices might be overlapping (ei. We can have a slice for the `car` sub-category that is inside the slice for the `transportation` category). This is fine since the dataset might have issue for only one of those categories.

However we can have some cases where the whole transportation category contains an issue (meaning that the `car` and other sub-categories would also contains this issue). That's why we want to filter the sub-slices from the scan report in order to improve it. 

I think you can start by having a look at the `PerformanceDetector` (`giskard.scanner.performance.performance_bias_detector.py`)
Repository: Giskard-AI/giskard-oss
Issue: Scan: Add a robustness detector to the scan that perturbs numerical values
Root cause: Unimplemented Feature Gap

## 🚀 Feature Request

Add a robustness detector to the scan that perturbs numerical values.

The scan should be able to a set of issues that capture the *minimum amount* of perturbation (lying in the bounds of the feature distribution) needed on a single numerical feature to:
- (a) change the predicted label (classification)
- (b) change the prediction by an amount that exceeds a certain threshold (regression)

## 🔈 Motivation

Currently the scan does not have any numerical perturbation.


Hey @kevinmessiaen! This seems to be a duplicate of #1847 
PS: I'd love to contribute to the tool! I'll be on the lookout for new issues/improvements :)

@pranavm7 Hey, it's not exactly the  same. One is for numerical values and the other is for categorical ones which differs a bit.

We would be happy to have you contribute on this tool, do you have any improvement ideas in mind?

@kevinmessiaen  I would like to work on this if this is still open. 

@Kranium2002 Sure we appreciate that, I assigned you to the issue. Let me know if you have some questions or need some help!

I would be working on adding a numerical perturbation detector to test model robustness by tweaking numerical features and seeing how much the model's predictions shift by around 1 %. For classification models, it'll flag cases where the predicted label changes, and for regression, it'll detect when predictions differ beyond a threshold (like 5%). I'll integrate this into the existing framework so it reports any significant sensitivity issues. Plus, I'll build out tests to ensure it's flexible across model types and datasets.

PS: Do I make the thresholds of 1 and 5 % editable by the user or do I keep them fixed?
Your thoughts? @kevinmessiaen 

Working on this in #2040
Repository: Giskard-AI/giskard-oss
Issue: Scan: Add a robustness detector to the scan that perturbs categorial values
Root cause: Unimplemented Feature Gap

## 🚀 Feature Request

Add a robustness detector to the scan that perturbs categorial values.

The scan should be able to a set of issues that capture the perturbations needed on a single categorial feature to:

(a) change the predicted label (classification)
(b) change the prediction by an amount that exceeds a certain threshold (regression)

## 🔈 Motivation

Currently the scan does not have any categorial perturbation.
<!-- Add any other context or screenshots about the feature request here. -->


Is this issue still active ? I would want to contribute to this issue 

@kevinmessiaen  I let you guide there, this seems easy to add, and a great idea of contribution!

Hello @ChatBear 

Yes this is still an active issue, I can assign you to it. We would be grateful to have your contribution, let me know if you have question about this.

Thanks, i'll try to contribute, i'll need a bit of time to understand the repo, after that i'll try to post PR 

Hello, i have few questions about the issue. 

What kind of pertubations do you except ? I was thinking of change the feature column with a probability of 0.1 (chosen arbitrary). 

And do i need the create another detector from scratch, or i can use a detector from BaseTextPerturbationDetector ? 

And i tried to create a branch, and i can't push in my own branch (i forked the repo but i am having trouble to create the pull request, i am kinda of new in open source so i apologize in advance if this question is inappropriate). 

Hello, 

The perturbation should be on categorical feature. It should only perturb on column of the dataset, the goal is to ensure that the model isn't too sensitive to noise. In this case the probability is not necessary since we want to test that the result isn't impacted when the value change. (it makes sense in text where we have typo rate for example).

Example is having a `breed` category with values potential values: `['Labrador', 'Husky', 'Beagle', ...]. The idea is to switch all `Labrador` value to any other breed and so on.

It won't work to reuse `BaseTextPerturbationDetector` since it cast column as `str` but we can have numerical categories for example. But you can inspire from it.





Ok, thanks i can continue