Repository: Giskard-AI/giskard-oss
Issue: [GSK-896] Better error handling
Root cause: Validation Gap

28/03/2023 (webedia)

* \[x\] **solved in** [**https://github.com/Giskard-AI/giskard/pull/911/commits/915e8a7a5936082a9e2068fab9b57f22f44614e9**](https://github.com/Giskard-AI/giskard/pull/911/commits/915e8a7a5936082a9e2068fab9b57f22f44614e9) --  `Dataset` initialization, the function signature just has one mandatory argument, but it actually requires at least one of the optional parameters (e.g. `cat_columns`, `column_types`). There should be a `ValueError` that was not raised and the user had no idea why he could not create a dataset with just `ds = Dataset(df)`. **This error is caused because one of the dataframe columns is of type** `list` (currently not supported by Giskard!) The user has a list column but they transforms it in the preprocessing before passing it to the model. User has columns of these types:
  * `list[int]`
  * `list[dict]`
* \[x\] **solved in** [**https://github.com/Giskard-AI/giskard/pull/911/commits/c172d7e4836506e813cdfd29b25d01bc40109dad**](https://github.com/Giskard-AI/giskard/pull/911/commits/c172d7e4836506e813cdfd29b25d01bc40109dad) -- `validate_model` user passes their model and not the Giskard model (→ should add an error message)
* \[x\] **solved with** [**https://github.com/Giskard-AI/giskard/pull/882**](https://github.com/Giskard-AI/giskard/pull/882) -- add an error message by checking the clf.**module** if it unmatch the wrapper class.
* \[x\] solved in [https://github.com/Giskard-AI/giskard/pull/911/commits/3644c49443540c9d1806c4bedbade7917f3a9513](https://github.com/Giskard-AI/giskard/pull/911/commits/3644c49443540c9d1806c4bedbade7917f3a9513) -- Preprocessing function taking 2 arguments (*df, blacklist* in this case): When the data preprocessing function is wrong there is no error message
  * check number of inputs of the preproc func and postproc func

30/03/2023 (webedia)

* [ ] The message error of the test is not clear -- @jmsquare do you have more info on this one?, @kevinmessiaen could you take care of it?
* [ ] unclear error when type is not good for `add_test`: It seems that you passed a `TestResult` to the test, please pass a test function reference and the parameters separately instead of calling the test function directly -- @kevinmessiaen could you take care of it?
* [X] solved in [https://github.com/Giskard-AI/giskard/pull/911/commits/30c2bacd47bb8dcaacf538668a388aacd07d7f7a](https://github.com/Giskard-AI/giskard/pull/911/commits/30c2bacd47bb8dcaacf538668a388aacd07d7f7a) -- `Target column not available` is not clear, have an explanation afterward on how to setup the target
* [X] solved in [https://github.com/Giskard-AI/giskard/pull/911/commits/6016fb09d4ed5e50f3e2b62cf1ef4244e0951795](https://github.com/Giskard-AI/giskard/pull/911/commits/6016fb09d4ed5e50f3e2b62cf1ef4244e0951795) -- When creating Giskard Dataset, we should have clear warning if target is not provides
* [X] solved in [https://github.com/Giskard-AI/giskard/pull/911/commits/bf104b90e74f525979d46de0c0fe23d77847bb9a](https://github.com/Giskard-AI/giskard/pull/911/commits/bf104b90e74f525979d46de0c0fe23d77847bb9a) -- When creating Giskard Dataset, we should do a check if the classification_label value are not the same
* [X] solved in [https://github.com/Giskard-AI/giskard/pull/911/commits/e0100ecde098bb225e5bb069f71d17ff2d598d49](https://github.com/Giskard-AI/giskard/pull/911/commits/e0100ecde098bb225e5bb069f71d17ff2d598d49) -- The error message “values in prediction label does not match with ..” should be completed by “provided in the model”
* [X] already [implemented](https://github.com/Giskard-AI/giskard/blob/21ab322aaf99ba7ef94be9d211c31ec58d538424/python-client/giskard/datasets/base/\__init_\_.py#L56-L57) -- userwarnings about the when the user does not provide it in `column_types`
* [X] the error message is quite clear as it is (see below) -- project_key cannot contain `-` or special characters. e.g. `spam-detection` didn’t work.

![](https://uploads.linear.app/5cb5c438-2552-41c0-89ac-b08d19e80d42/981fd1a9-628b-47c0-bf94-211e8ab1cc80/7e49cbaa-984e-4be9-9dc8-7bcd0dce547c)

* [ ] Our drift tests are wrong, because it should have returned an error since the classification_labels between ground truth and prediction are not the same (”1.0” and 1) -- @jmsquare could you elaborate on this one please?

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-896](https://linear.app/giskard/issue/GSK-896/better-error-handling)</sub>

For the comment:
`Our drift tests are wrong, because it should have returned an error since the classification_labels between ground truth and prediction are not the same (”1.0” and 1) -- @jmsquare could you elaborate on this one please?`

@rabah-khalek when you're going to modify the tests, we'll need to think about error handling. For instance, for prediction drift tests, what do we do if the intersection of `unique(y_pred)` and `unique(y_true)` is empty? Today we don't handle it qith an error message.