zhiminy/EvalEng huggingface dataset:
Repository: Giskard-AI/giskard-oss
Issue: [GSK-897] The arg classification_label in tests is not clear
Root cause: Validation Gap

people tend to put the name of the target column instead.

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-897](https://linear.app/giskard/issue/GSK-897/the-arg-classification-label-in-tests-is-not-clear)</sub>

A solution to me is to make the error message clearer. 

<!-- From jmjm on Linear -->

Currently if I do this:

```
from giskard import test_drift_prediction_ks
result = test_drift_prediction_ks(reference_slice=my_dataset,
                                  actual_slice=my_dataset,
                                  model=my_model,
                                  classification_label='default',
                                  threshold=0.5).execute()
```

the error message is quite clear:

```
AssertionError: "default" is not part of model labels: 0,1
```

I can add an error to check that if the user gives a `classification_label == data.target` I give them a clarification and tell them what to do…

<!-- From rabah on Linear -->

I'll wait on the migration of all tests from `giskard/ml_worker/testing/` to `giskard/ml_worker/testing/tests` by @kevinmessiaen 

<!-- From rabah on Linear -->

Ok now if I have:

```
from giskard import test_drift_prediction_ks
result = test_drift_prediction_ks(reference_slice=my_dataset,
                                  actual_slice=my_dataset,
                                  model=my_model,
                                  classification_label='default',
                                  threshold=0.5).execute()
```

I get the following error instead:

```
AssertionError: By "classification_label", we refer to one of the values: [0,1] and not the target: "default". Please re-assign this argument.
```

This error:

```
AssertionError: "default" is not part of model labels: 0,1
```

remains for when `classification_label!=target and classification_label not in classification_labels`

<!-- From rabah on Linear -->

solved in https://github.com/Giskard-AI/giskard/pull/924/commits/d3f445adbfda376ddb0f7a7e48f755684245b96e
![Screenshot 2023-04-12 at 14 54 34](https://user-images.githubusercontent.com/32709181/231463722-cbbde1e1-d5f8-4252-858d-432fe0e7a406.png)
Repository: Giskard-AI/giskard-oss
Issue: [GSK-898] Test output is not clear
Root cause: Documentation Deficiency

many test output keys (like unexpected_percent\_... ) is not clear.

We should also implement the `__repr__`  method to improve the visibility

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-898](https://linear.app/giskard/issue/GSK-898/test-output-is-not-clear)</sub>

The output of tests will change a lot once we introduce the debugging function.

<!-- From jmjm on Linear -->

@jmsquare How do you think about some test result that can be displayed like that?

![image.png](https://uploads.linear.app/5cb5c438-2552-41c0-89ac-b08d19e80d42/510248a1-d125-4520-9419-ee56c5f11a11/c6ebe758-b2b6-44fb-9838-61c0f1c583df)

![image.png](https://uploads.linear.app/5cb5c438-2552-41c0-89ac-b08d19e80d42/f05ec591-9f7a-4a15-8674-e3fb193cce4f/24168fee-09cb-4578-95ab-dd6906f6168d)

<!-- From kevin on Linear -->

That's nice! I don't think it's necessary to put all the path: giskard.ml_worker.testing.tests.performance… Maybe putting just test_f1 and the identifier of the tests (if there is any), it's ok!

<!-- From jmjm on Linear -->
Repository: Giskard-AI/giskard-oss
Issue: [GSK-899] The name attribute of the test is not clear
GitHub labels: user-test

webedia



<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-899](https://linear.app/giskard/issue/GSK-899/the-name-attribute-of-the-test-is-not-clear)</sub>

We ask a string (ex: "f1") as an argument of the add_test function. This is not clear at all

<!-- From jmjm on Linear -->

@jmsquare I think that since this argument is optional and "advanced". We should remove it from the basic example  in the documentation and provide a better description on the API reference

<!-- From kevin on Linear -->

Completely agreed! Can you remove it from the doc? @kevinmessiaen
Repository: Giskard-AI/giskard-oss
Issue: [GSK-900] issue with classification_labels
Root cause: Validation Gap

![Screenshot 2023-03-30 at 16.20.27.png](https://uploads.linear.app/5cb5c438-2552-41c0-89ac-b08d19e80d42/672d51e4-8fc4-4424-bf02-296a39b68f33/9a29a851-3384-4d82-95b0-95333ec550c8)

* issue with classification_labels provided to the model

  ```
  posts['label'] = posts['label'].astype("int32")
  
  ```
* could be that the ones provided are `str` and the one inferred from `classes_` are int? → add a unit-test on this

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-900](https://linear.app/giskard/issue/GSK-900/issue-with-classification-labels)</sub>

There's indeed currently this issue:

```
my_df = df.copy()
new_map = {"Default": 0, "Not default": 1}
my_df['default'] = my_df['default'].map(new_map)

my_dataset = Dataset(
    df=my_df,
    target="default",
    column_types=input_types,
)

my_model = SKLearnModel(
    clf=clf,
    model_type="classification",
    feature_names=list(input_types),
    classification_threshold=0.5,
    classification_labels=[0, 1],
)

from giskard.core.model_validation import validate_model
validate_model(my_model,my_dataset)
```

```
Hint: "Your target variable values are numeric. It is recommended to have Human readable string as your target values to make results more understandable in Giskard."
ValueError: Values in default column are not declared in classification_labels parameter: {'0', '1'}
```

which is due to this line in `model_validation.py`:

```
 target_values = target_values if is_string_dtype(target_values) else [str(label) for label in target_values]
```

<!-- From rabah on Linear -->

solved here: https://github.com/Giskard-AI/giskard/pull/924/commits/4148b2c19b9d95d9f64d07098e89e6ace3b11ef8

I removed the `str` casting to classification label(s), and this issue is now solved in https://github.com/Giskard-AI/giskard/pull/924/commits/b4c7fe6c358a47662d992b2f4433ff6de26eb9b8
Repository: Giskard-AI/giskard-oss
Issue: [GSK-901] test related issues (Webedia)
Root cause: Documentation Deficiency

* User didn’t intuitively use test function object as a first argument of `add_test`, he wanted to call it directly
* it wasn’t clear that we have 2 modes of creating tests: function and class based

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-901](https://linear.app/giskard/issue/GSK-901/test-related-issues)</sub>

fixed by the new `GiskardTestMethod` implementation that allows to call `add_test(my_test(arg1=arg))` instead of `add_test(my_test, arg1=arg)`

<!-- From Andrey Avtomonov on Linear -->
Repository: Giskard-AI/giskard-oss
Issue: [GSK-902] UI related issues
GitHub labels: user-test

webedia

* Giskard pull of the dev version does not work (tag api-v2) → wsl2? (should test on Windows)
* L’ui peut être intéressant pour le monitoring est voir l’ensemble des tests qui passent ou pas
* He wants to logs in the UI the execution of the suite in the notebook
* uploading a previously created license is not obvious at all atm (via “upload it”). Nicolas was tempted to create a new license.
* We should first upload without giving a project name

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-902](https://linear.app/giskard/issue/GSK-902/ui-related-issues)</sub>

Since all the points (except for the 4th ones) are not really relevant today. I mark this card as Done and create a new card on license

<!-- From jmjm on Linear -->
Repository: Giskard-AI/giskard-oss
Issue: [GSK-924] Refactorings in python-client
GitHub labels: enhancement

new-upload-api

ai-test-v2



<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-924](https://linear.app/giskard/issue/GSK-924/refactorings-in-python-client)</sub>

@andreybavt, since as you mentionned, we're moving things to the programmatic realm, it makes sense to elevate some scripts from `python-client/giskard/ml_worker` to `python-client/giskard`. 

I would mainly like to do the following refactorings:
- `python-client/tests` --> `python-client/unit_tests`
- `python-client/giskard/ml_worker/testing` --> `python-client/giskard/tests`

together with (this is the current structure in [feature/new-upload-api-automodel](https://github.com/Giskard-AI/giskard/tree/feature/new-upload-api-automodel)):
```
python-client/giskard
├── __init__.py
├── cli.py
├── cli_utils.py
├── core
│   ├── __init__.py
│   ├── core.py
│   ├── dataset_validation.py --> python-client/giskard/datasets/.
│   ├── model_validation.py --> python-client/giskard/models/.
│   └── validation.py
├── datasets
│   ├── __init__.py
│   └── base
│       └── __init__.py
├── ml_worker
│   ├── __init__.py
│   ├── core
│   │   ├── model_explanation.py --> depends on whether Matteo will use it in scan or not. 
│   │   ├── suite.py --> python-client/giskard/tests/.
│   │   ├── test_result.py --> python-client/giskard/tests/.
│   │   └── test_runner.py --> python-client/giskard/tests/.
│   └── ml_worker.py
├── models
│   ├── __init__.py
│   ├── base
│   │   └── __init__.py
│   ├── catboost
│   │   └── __init__.py
│   ├── huggingface
│   │   └── __init__.py
│   ├── pytorch
│   │   └── __init__.py
│   ├── sklearn
│   │   └── __init__.py
│   ├── tensorflow
│   │   └── __init__.py
│   └── utils.py
├── path_utils.py
└── settings.py
```
WDYT?

@rabah-khalek-khalek this is done now, right?

<!-- From Andrey Avtomonov on Linear -->

No, I haven't worked on it

semi-done with v2
Repository: Giskard-AI/giskard-oss
Issue: [GSK-920] Display some information about datasets (number of entries)
Root cause: Unimplemented Feature Gap

* [ ] showing some dataset stats (like the number of entries, columns, etc.) on the frontend projects/datasets
* [ ] showing the name of the dataset in the inspection window

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-920](https://linear.app/giskard/issue/GSK-920/ui-small-improvements)</sub>

Hi @rabah-khalek I would like to work in this issue. 

Hey @shamim-io I've assigned the issue to you. Please open a PR when you've got some work to show, and don't hesitate to ask for help here or on our Discord !
Repository: Giskard-AI/giskard-oss
Issue: [GSK-890] "Target column is not available" message should include "in the dataset"
GitHub labels: user-test

webedia

Maybe also an example of how to specify a target for a dataset

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-890](https://linear.app/giskard/issue/GSK-890/target-column-is-not-available-message-should-include-in-the-dataset)</sub>

duplicate of an issue that was solved here: [https://github.com/Giskard-AI/giskard/commit/30c2bacd47bb8dcaacf538668a388aacd07d7f7a](https://github.com/Giskard-AI/giskard/commit/30c2bacd47bb8dcaacf538668a388aacd07d7f7a)

<!-- From rabah on Linear -->
Repository: Giskard-AI/giskard-oss
Issue: [GSK-945] Generate proto stubs when installing giskard python library
Root cause: Architectural Constraint

Run whatever is executed in `python-client/build.gradle.kts:55` but during `pip install`

ideally this should be described in `pyproject.toml`

\*\*\*

Tested with @andreybavt : we can use `setuptools` for the build, migrate to standard PEP 621 for the `pyproject.toml`. We then drop poetry for development since it is not compatible with PEP 621.

### Consequences

* we will not need the gradle build to generate the whl, meaning that we can install directly from the git subdirectory
* this will drop the poetry dependency for the package build, hopefully giving less problems with installation (setuptools is often already installed)

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-945](https://linear.app/giskard/issue/GSK-945/generate-proto-stubs-when-installing-giskard-python-library)</sub>

The problem when installing from VCS is a `pip` bug (see previous comment). It skips the build because it somehow wrongly estimates that the build was already done.

A quick workaround is to force the reinstall (which is good default anyway for dev builds):

```
$ pip install "git+https://[…]" --force-reinstall --no-deps
```

<!-- From mattbit on Linear -->

[https://github.com/pypa/pip/issues/10095](https://github.com/pypa/pip/issues/10095)

<!-- From mattbit on Linear -->
Repository: Giskard-AI/giskard-oss
Issue: [GSK-953] Where is giskard installation path on mac
Root cause: Documentation Deficiency

## Checklist

<!-- Mark with an `x` all the checkboxes that apply (like `[x]`) -->

- [ ] I've searched the project's [`issues`](https://github.com/Giskard-AI/giskard/issues?q=is%3Aissue).

## ❓ Question

<!-- What is your question -->

Where is giskard installation path on mac



<sub>[GSK-953](https://linear.app/giskard/issue/GSK-953/where-is-giskard-installation-path-on-mac)</sub>

Hi @Tonykaynoni ,

Giskard has 2 main components: server and ML Worker.

The server run in Docker, it creates 2 Docker volumes to store required data, to identify these volumes you may run a following command:
`docker volume ls | grep -E 'giskard|app-db-data'`


In case you start an ML Worker on your mac (by running `giskard worker start`) there will be a local directory created in `~/giskard-home` that will contain files related to the ML Worker operation
Repository: Giskard-AI/giskard-oss
Issue: [GSK-955] doc related issues (Altaroad)
GitHub labels: d:easy

user-test

Altaroad

* a clear link to Test Catalog from “Create custom test suite” is missing!
  * He does not find the test catalog link
  * He looks for documentation on Github README

deprecated or already done:

* It wasn’t clear to him that the feature_names should be the raw columns names and NOT the preprocessed names
* The section “Create custom test” in the doc is a bit misleading. We can remove “Custom”
* He finds that it is too oriented toward classificiation
* He has no clue about the test he wants to implement. Having clear tags about “regression” int the test catalog would have helped
* \~the difference between model and custom model is hard to get\~
* \~The word “Create” a model is not correct. I got confused because he already has a model created\~
* \~we should add a comment in the doc to avoid providing “classification_labels” if it’s a regression model\~
* putting `example` before `main parameters` in the doc made him completely missing the explanations (”j’ai du regarde ca”). He copy paste the example before reading the doc
  * Yet, examples are more important than the reference for him
    * if he doesn’t find an example snippet usually, he doesn’t even try it.
* Maybe we can remove feature_names from the code snippet? But we need to have a clear error message if we finally need that the user specify the feature_names
* for him, feature_names are the name of features used by the model, It not clear that feature_names is the parameter of the model

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-955](https://linear.app/giskard/issue/GSK-955/doc-related-issues-altaroad)</sub>

@rabah-khalek-khalek can you update this with the latest changes?

<!-- From alex on Linear -->

Yep, this is part of my assigned issues, i tried first to prioritize the code based one. I'll work more on the doc tomorrow!
Repository: Giskard-AI/giskard-oss
Issue: [GSK-954] model related issues
Root cause: Documentation Deficiency

* [X] clf issues -- solved in [https://github.com/Giskard-AI/giskard/pull/924/commits/d171ca2de05263fd6f7abf65fc7d77d4b5106771](https://github.com/Giskard-AI/giskard/pull/924/commits/d171ca2de05263fd6f7abf65fc7d77d4b5106771) and [https://github.com/Giskard-AI/giskard/pull/924/commits/c7701dfe23a3ab94226bdb8e77630e7251541277](https://github.com/Giskard-AI/giskard/pull/924/commits/c7701dfe23a3ab94226bdb8e77630e7251541277)
  * [X] My_clf : he does not know f it covers the transformation of not
  * [X] `clf` refers to a classifier — so it wasn’t intuitive to use in the case of regression
  * [X] Clf: he does not know what this is (also, he is working with a regression model, so clf makes no sense) He tried to pass a `reg` parameter (”for me clf is a classifier”).
    * [X] after a [discussion](https://giskardai.slack.com/archives/C050NFGC3M1/p1680869176557859?thread_ts=1680864742.795389&cid=C050NFGC3M1) on slack, the solution is to have `model` as argument, and `wrap_model` and `wrap_dataset` as factory method instead.
* [X] For the automatic type inference, how can the user know what is the type that Giskard inferred -- solved in [https://github.com/Giskard-AI/giskard/pull/924/commits/42261f422c881df3b092e4860d182499d1eab69d](https://github.com/Giskard-AI/giskard/pull/924/commits/42261f422c881df3b092e4860d182499d1eab69d)

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-954](https://linear.app/giskard/issue/GSK-954/model-related-issues)</sub>

This is now the message a user sees when he uses `wrap_model`: 
![Screenshot 2023-04-14 at 11 51 57](https://user-images.githubusercontent.com/32709181/232011560-bf8c3e30-8640-49da-bb5c-1e5caaf7a9bd.png)
Repository: Giskard-AI/giskard-oss
Issue: [GSK-956] test related issues (Altaroad)
Root cause: Documentation Deficiency

* [X] He doesn’t seem to be familiar with typings, he wrote (`test_X(model: my_model, actual_slice: my_dataset)` instead of `test_X(model= my_model, actual_slice= my_dataset)`) => Allowing args should help
* [X] tests do not work with `args` only `kwargs`
* [X] it’s not clear which test gets executed where (their order) after `suite.run()`
* [X] It’s hard to loop over the test suite results, since the keys contain `uuid`s
* [X] Suite test result is not clear, we need to provide some UI maybe.
* [X] \~Does not know how to get a “name” to select a given test. He is looking for a name or a tag to select programmatically the test object.git\~ => Use list instead of dict
* [X] He doesn’t use often auto-completion
* [X] \~he want to add the\~ `~test_identifier~` \~to a test instance rather than to the\~ `~add_test~` \~method of\~ `~Suite~`
* [X] Metric is the same for two test executions ⇒ bug in the suite? (same test function with different parameters might be wrongly identified in run method of suite)
* [X] When a user instantiate a test, we need to throw error message if the parameter that are passed are not the good one (test_identifier)
* [X] test suite uses `save` but model and dataset use `upload` — we should use `upload` for everything
* [X] He did not understand first that Suite() is a function that can contain variables
* [X] It’s not intuitive for him how suite input are defined ⇒ example seems to help him
* [X] It’s not clear if `run` `kwarg` override test inputs or not
* [X] it’s not clear which inputs were used by the tests executed by the test suite. In the results of test execution we should see inputs that were used to run the test
* [X] He think a details option for run would be nice to see detailed info about the execution (param used, …)
* [X] \~Would like to allow others to download the test suites from Giskard, for collaboration\~
* [X] `@test` on function give the following error: `missing 1 required positional argument: func`
* [X] Autocompletion generated after using `@test` is not clear

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-956](https://linear.app/giskard/issue/GSK-956/test-related-issues-altaroad)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-958] Standardisation of tests signature
Root cause: Unimplemented Feature Gap

After a discussion with @jmsquare and @kevinmessiaen we decided to do:

* refactor: args like df, actual_slice, reference_slice → dataset, actual_dataset, reference_dataset
* add a slice_function as arg in all tests (requires also a change in frontend, and backend)
* for disparate_impact: keep protected/unprotected but add \*\_slice_function suffix.
* refactor perturbation_dic -> transformation(requires the creation of transformation class)

To be done once the implementation of programmatic filters by @andreybavt is more or less stable.

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-958](https://linear.app/giskard/issue/GSK-958/standardisation-of-tests-signature)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-979] Progress bar for the scan
GitHub labels: scan

Andrey: it’d be nice to add a progress bar for the scan (tqdm for example) to let a user know that something is happening.

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-979](https://linear.app/giskard/issue/GSK-979/progress-bar-for-the-scan)</sub>

`tqdm` not very nice for now because it’s very limited in handling undetermined progress.

Tried with spinners like `halo` and `yaspin` but they are not super reliable with different environments (term, ipython, jupyter, etc.).

For now, I just provide simple textual output to stdout.

<!-- From Matteo Dora on Linear -->
Repository: Giskard-AI/giskard-oss
Issue: [GSK-980] Ambiguity of options for cat_columns, column_types, infer_column_types
Root cause: Documentation Deficiency

When he create the dataset, he does not understand the thee different option. In the documentation we should explain in which case a specific option is preferable

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-980](https://linear.app/giskard/issue/GSK-980/ambiguity-of-options-for-cat-columns-column-types-infer-column-types)</sub>

@mattbit if we decorelate scan from the accuracy of infer_column_types, we can then make:

* infer_column_types the default of `Dataset` 
* `cat_columns` and `column_types` as optional in case the user wants to enforce it.

<!-- From rabah on Linear -->

solved in [https://github.com/Giskard-AI/giskard/pull/965](https://github.com/Giskard-AI/giskard/pull/965)

<!-- From rabah on Linear -->
Repository: Giskard-AI/giskard-oss
Issue: [GSK-981] Better output from the pydantic decorator validate_arguments
GitHub labels: user-test

CGI

* It’s not clear for that we expect a wrapped model `BaseModel` instead of a `clf`
  * The method `wrap_model` is confusing as it return a `BaseModel` , maybe a `WrappedModel` would be less confusing

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-981](https://linear.app/giskard/issue/GSK-981/better-output-from-the-pydantic-decorator-validate-arguments)</sub>

@rabah-khalek-khalek: is it solverd today? If yes, can you put the card as Done

<!-- From jmjm on Linear -->

Not done yet.

I didn't find the pydantic decorator to be super customizable, the other solution is to write our own decorator (which is tideous, a start is attempted here: [https://github.com/Giskard-AI/giskard/tree/task/pydantic_decorator](https://github.com/Giskard-AI/giskard/tree/task/pydantic_decorator) but not complete).

Maybe pydantic v2 would be already improved w.r.t the one we currently use ([https://docs.pydantic.dev/latest/blog/pydantic-v2/](https://docs.pydantic.dev/latest/blog/pydantic-v2/)).

<!-- From rabah on Linear -->
Repository: Giskard-AI/giskard-oss
Issue: [GSK-982] Integrate validate_model in scan
Root cause: Validation Gap

* Scan error are not clear when the parameters are set incorrectly
  * We should put a clear error message if the input type of validate_model is not correct.
    * ✅ + put afine message when it’s validated
  * we did not validate the model, we need type checking
  * is he passing the correct model object? we do not check that in `scan`

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-982](https://linear.app/giskard/issue/GSK-982/integrate-validate-model-in-scan)</sub>