zhiminy/EvalEng huggingface dataset:
Repository: Giskard-AI/giskard-oss
Issue: Issue with the old test API, test suite execution throws error if ref dataset not determined
Root cause: Validation Gap

### Issue Type

Bug

### Source

source

### Giskard Library Version

1.7.2

### Giskard Server Version

1.3.2

### OS Platform and Distribution

mac

### Python version

3.8

### Installed python packages

_No response_

### Current Behaviour?

```shell
File "Giskard_test.py", line 76, in <module>
    test_result = credit_scoring.execute_test_suite(
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.8/site-packages/giskard/client/project.py", line 162, in execute_test_suite
    self._update_test_suite_params(
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.8/site-packages/giskard/client/project.py", line 101, in _update_test_suite_params
    res = self._session.put("testing/suites/update_params", json={
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.8/site-packages/requests/sessions.py", line 647, in put
    return self.request("PUT", url, data=data, **kwargs)
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.8/site-packages/requests_toolbelt/sessions.py", line 64, in request
    return super(BaseUrlSession, self).request(
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.8/site-packages/requests/adapters.py", line 584, in send
    return self.build_response(request, resp)
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.8/site-packages/giskard/client/giskard_client.py", line 49, in build_response
    raise giskard_error
giskard.client.giskard_client.GiskardError: Internal Server Error: Cannot invoke "ai.giskard.domain.ml.Dataset.getId()" because the return value of "ai.giskard.domain.ml.TestSuite.getReferenceDataset()" is null
```


### Standalone code OR list down the steps to reproduce the issue

```shell
1. Create a project
2. upload a model and df
3. Go to the UI and create a test suite with only actual_ds
4. retrieve the test suite from the UI and try to execute, e.g.

    test_result = credit_scoring.execute_test_suite(
        test_suite_id=test_suite_id,
        model_id=model_id)
```


### Relevant log output

_No response_

closing as outdated
Repository: Giskard-AI/giskard-oss
Issue: A prediction_function returning an int isn't validated and passes the upload
Root cause: Validation Gap

A simple model like:
```
def model(df):
    return 5
```

passes the upload validation and causes inspector (both prediction and explanation) to break

Let us close this issue with the merge of https://github.com/Giskard-AI/giskard/pull/618 as it'll become irrelevant.

@rabah-khalek could you check that there's a unit test for it in the v2 branch?

Yes it's here: https://github.com/Giskard-AI/giskard/blob/9db3efc9918492c17333869b3dc6da852f0feb4a/python-client/tests/test_model_postprocess.py#L26
Repository: Giskard-AI/giskard-oss
Issue: Small datasets can't be viewed from UI
Root cause: Validation Gap

to reproduce:
- take a full `test_data.csv` from german credit scoring demo project
- upload it to Giskard
```
import pandas as pd
df = pd.read_csv('test_data.csv').head(2)
p.upload_df(df=df, column_types={c:'text' for c in df.columns}, name='2 lines')
```
- open it with dataset viewer

<img width="1173" alt="image" src="https://user-images.githubusercontent.com/7889985/216398037-05ba9f5a-db93-490d-a411-0844de77c7fa.png">
Repository: Giskard-AI/giskard-oss
Issue: Filter function is AI Inspect display "1/1" instead of "0/0" when there is no examples in filtering
Root cause: Algorithmic Error

When there is no data after the filter function (incorrect, borderline) : AI Inspect display: 1/1

Asked by Instant gaming

Instead of 0/0 we can have a nicer message that "No data matches the selected filter"

Great!
Repository: Giskard-AI/giskard-oss
Issue: Test if the deamon connection ml worker is stable after 12 hours
Root cause: Resource Mishandling

- DigitRE has to re start ml worker every day
- Same for Intant gaming

I added a heartbeat for an ML Worker tcp connection to prevent it from being closed. On our dev instance there doesn't seem to be any issue (screenshot taken on 06/02 15:24 - 2 days after the worker was started):

![image](https://user-images.githubusercontent.com/7889985/216996542-f3ce4abe-9141-421b-a541-78ddfddfcece.png)
Repository: Giskard-AI/giskard-oss
Issue: Test, for regression, if the prediction function returns a list or int, that the upload passes
Root cause: Validation Gap

For regression especially, we should not ask for numpy array as the output of model.

linked to https://github.com/Giskard-AI/giskard/issues/692

https://github.com/Giskard-AI/giskard/issues/692 is closed
Repository: Giskard-AI/giskard-oss
Issue: Possibility to create slices without name

### Issue Type

Bug

### Source

docker

### Giskard Library Version

1.7.0.

### Giskard Server Version

1.3.2

### OS Platform and Distribution

_No response_

### Python version

_No response_

### Installed python packages

_No response_

### Current Behaviour?

```shell
It is possible to create a slice without specifying its name. The name should be mandatory.
```


### Standalone code OR list down the steps to reproduce the issue

```shell
- Open a project and go to the model tab
- Inspect the model
- Click on slice to apply and then select create new slice
- Save while letting the slice name field blank
```


### Relevant log output

_No response_

Good catch, will fix asap!
Repository: Giskard-AI/giskard-oss
Issue: OSError: [Errno 30] Read-only file system: 'tmp_trainer' - external worker started as deamon
Root cause: Resource Mishandling

### Issue Type

Bug

### Source

docker

### Giskard Library Version

1.7.0

### Giskard Server Version

1.3.2

### OS Platform and Distribution

macOS Ventura 13.2

### Python version

3.8.16

### Installed python packages

transformers==4.26.0

### Current Behaviour?

```shell
When running Giskard as a daemon (-d) option, the error message appears, the error message does not appear when running without -d option
```


### Standalone code OR list down the steps to reproduce the issue

```shell
- Start an external ML worker using `giskard worker start -d`
- Upload `Sentiment_Analysis_for_Twitter_Data` project from giskard-examples
- Inspect the dataset and see the error logs

Then:
- stop the ML worker using `giskard worker stop`
- Start the ML worker using `giskard worker start`
- Inspect the dataset without error
```


### Relevant log output

```shell
2023-02-08 10:45:59,939 pid:13172 MainThread giskard.ml_worker.utils.error_interceptor ERROR    [Errno 30] Read-only file system: 'tmp_trainer'
Traceback (most recent call last):
  File "/Users/kevinmessiaen/.pyenv/versions/3.8.16/lib/python3.8/site-packages/giskard/ml_worker/utils/error_interceptor.py", line 48, in wrapper
    res = await loop.run_in_executor(pool, behavior, request, context)
  File "/Users/kevinmessiaen/.pyenv/versions/3.8.16/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/kevinmessiaen/.pyenv/versions/3.8.16/lib/python3.8/site-packages/giskard/ml_worker/server/ml_worker_service.py", line 224, in runModel
    prediction_results = model.run_predict(dataset)
  File "/Users/kevinmessiaen/.pyenv/versions/3.8.16/lib/python3.8/site-packages/giskard/ml_worker/core/model.py", line 40, in run_predict
    raw_prediction = self.prediction_function(df)
  File "/Users/kevinmessiaen/work/giskard-examples/venv/lib/python3.8/site-packages/giskard/client/project.py", line 437, in <lambda>
    return lambda df: prediction_function(df[feature_names])
  File "/var/folders/nf/w2h_y58n4sxdyjmq4qlwcxym0000gn/T/ipykernel_3030/1249985334.py", line 10, in predict
  File "/Users/kevinmessiaen/.pyenv/versions/3.8.16/lib/python3.8/site-packages/transformers/trainer.py", line 505, in __init__
    os.makedirs(self.args.output_dir, exist_ok=True)
  File "/Users/kevinmessiaen/.pyenv/versions/3.8.16/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: 'tmp_trainer'
```


fixed in 57a90f0dbc931bb294e56b0c4b2595646e882b43
Repository: Giskard-AI/giskard-oss
Issue: Filtering doesn't work on texts containing `\n` caracters
Root cause: Algorithmic Error

Currently, we iterate over the lines read by BufferedReader:

https://github.com/Giskard-AI/giskard/blob/e57911a8f8a85c9c2c0441451e93b7838cfec47b/backend/src/main/java/ai/giskard/service/ml/MLWorkerService.java#L239


This doesn't handle the case when the text inside a table cell also contains `\n` characters.

To reproduce:
- Create an inspection in ENRON project
- use a filter like:
```
def filter_row(row):
    return len(row.Content) > 0
```

The results are the following 
![image](https://user-images.githubusercontent.com/7889985/217525010-732ef8f6-26e0-4ed7-8912-b7502b977486.png)


We should probably rely on Apache commons CSV to read records:
```java
final CSVFormat csvFormat = CSVFormat.Builder.create()
        .setHeader(HEADERS)
        .setAllowMissingColumnNames(true)
        .build();
final Iterable<CSVRecord> records = csvFormat.parse(reader);

```

Related issue:
Backend errors aren't reported to the frontend when they occur during the filtering
Repository: Giskard-AI/giskard-oss
Issue: Add a spinner after the "Next" is clicked on a project import modal
GitHub labels: enhancement

When the project is large, visually nothing happens after the Next button is clicked. We should disable the button and show the spinner

![image](https://user-images.githubusercontent.com/7889985/217532993-c5542778-1c03-467c-a579-641ab189015d.png)

Same for the Import button too
Repository: Giskard-AI/giskard-oss
Issue: Some automatic tests fail when providing prod as actual_ds and test/train as reference_ds
Root cause: Validation Gap

### Issue Type

Bug

### Source

docker

### Giskard Library Version

1.7.0

### Giskard Server Version

	1.3.2

### OS Platform and Distribution

mac

### Python version

_No response_

### Installed python packages

_No response_

### Current Behaviour?

```shell
Some automatic tests fail when providing prod as actual_ds and test/train as reference_ds.
the issue is that we're currently creating automatic tests that require both actual_ds and reference_ds without checking if one of them is "prod" (no target) or not.
```


### Standalone code OR list down the steps to reproduce the issue

```shell
create automatic tests with prod as actual_ds --> all tests work
create automatic tests with prod and train or test --> some tests fail
```


### Relevant log output

_No response_
Repository: Giskard-AI/giskard-oss
Issue: Generate an invite for an user show [object Object]

### Issue Type

Bug

### Source

source

### Giskard Library Version

1.7.0

### Giskard Server Version

1.3.2

### OS Platform and Distribution

_No response_

### Python version

_No response_

### Installed python packages

_No response_

### Current Behaviour?

```shell
When I generate an invitation link for an user, the following message appears the first time: `[object Object]`, the second time it works as expected.
```


### Standalone code OR list down the steps to reproduce the issue

```shell
- Activate user Authentication
- Go to settings then tab users and click invite
- Click generate
```


### Relevant log output

_No response_
Repository: Giskard-AI/giskard-oss
Issue: Create a demo instance of Giskard that is restarted every day
Root cause: Unimplemented Feature Gap

Problem: all our notebook examples are often executed using Colab. But there is no way to upload a model created in Colab in Giskard if Giskard is installed locally. For example, students do not often install Giskard in the Cloud and uses usually Colab. What could be a way for them to upload a model.

An easy way would be to have an open instance (demo.giskard.ai) that is accessible from Colab. This instance could be restarted and shut down every day.

Great idea! I would not call it demo, as we already have a demo.giskard.ai instance, which serves for internal demo. We need to handle that one internally.

What you explain should probably be called a sandbox.

However there are very significant legal risks, because if we open a giskard instance to the world, it means we host data. That brings a whole set of requirements with GPDR, data security etc. Even if these may be students using this, we cannot control what data they store there.

I would suggest you engage with our law firm to scope the risks and what could be done to alleviate these risks. 

I would also advise we spend more time on the specification phase of this problem. Some questions:

- Are we sure that the solution of one server that is restarted everyday would fit the requirements? It means that people would lose their work.
- What is the maintenance cost associated with managing this open instance?
- Is this solution required if we offer a way to host on HF spaces?



This is solved by the new HF spaces Hub instance ✅
Repository: Giskard-AI/giskard-oss
Issue: Investigate the possibility to use shap instead of Lime 
Root cause: Unimplemented Feature Gap

The problem : Lime is not robust because its masking process is totally random and does take into account the model objects (tokenizer etc).

Since we created an upload process specific to model, we can investigate how to use  shap.Explainer instead of LIme. For HF model, the advantage would be that the tokenizer inside the model would be better taken into account to create the masking process of the explanation

For more info, see https://shap.readthedocs.io/en/latest/example_notebooks/text_examples/sentiment_analysis/Emotion%20classification%20multiclass%20example.html
Repository: Giskard-AI/giskard-oss
Issue: [GSK-541] Migrate vue component declaration from @Component to composition API
GitHub labels: good first issue

yt

p:normal

d:easy

[GSK-351](https://giskard.youtrack.cloud/issue/GSK-351)

## Newly created components should ONLY use composition API of Vue 2.7

### Migration:

`vue-property-decorator` library doesn't support Vue 3, so in order to migrate to Vue 3 we have to migrate component definition from @Component to composition API

* [ ] Admin.vue
* [X] AdminUsers.vue
* [X] App.vue
* [ ] AuthPortal.vue
* [X] ButtonModalConfirmation.vue
* [ ] CreateUser.vue
* [ ] CustomInspectionFilter.vue
* [X] Dashboard.vue
* [ ] DatasetSelector.vue
* [X] Datasets.vue
* [X] EditUser.vue
* [ ] EditableLabel.vue
* [ ] ErrorToast.vue
* [X] FeedbackDetail.vue
* [X] FeedbackList.vue
* [ ] FeedbackPopover.vue
* [ ] InspectionFilter.vue
* [ ] Inspector.vue
* [X] InspectorLauncher.vue
* [ ] InspectorWrapper.vue
* [X] InviteUsers.vue
* [X] Login.vue
* [ ] Main.vue
* [X] MessageReply.vue
* [ ] ModelSelector.vue
* [X] Models.vue
* [ ] MultiSelector.vue
* [X] OverlayLoader.vue
* [X] PasswordRecovery.vue
* [ ] PredictionExplanations.vue
* [ ] PredictionResults.vue
* [X] Project.vue
* [ ] ProjectOverview.vue
* [X] ProjectsHome.vue
* [X] ResetPassword.vue
* [ ] ResultPopover.vue
* [ ] RouterComponent.vue
* [X] Signup.vue
* [ ] Start.vue
* [ ] TestCreateModal.vue
* [ ] TestEditor.vue
* [ ] TestSuite.vue
* [ ] TestSuiteCreateModal.vue
* [ ] TestSuiteSettings.vue
* [ ] TestSuites.vue
* [ ] Tests.vue
* [X] TextExplanation.vue
* [ ] UploadButton.vue
* [X] UserProfile.vue
* [X] UserProfileEditPassword.vue

Project: Giskard
Reporter: aavtomonov
Created: 2022-11-13

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-541](https://linear.app/giskard/issue/GSK-541/migrate-vue-component-declaration-from-component-to-composition-api)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-703] Make it possible to disconnect an ML Worker from the UI
GitHub labels: enhancement

It should be possible to disconnect an ML Worker (or workers once we support multiple ones connected) from the UI. Currently if the connection between backend and ML Worker fails, the worker keeps trying to reconnect until it succeeds.
In order to stop ML Worker gracefully, we could send a "stop" command that would lead to the worker process exit

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-703](https://linear.app/giskard/issue/GSK-703/make-it-possible-to-disconnect-an-ml-worker-from-the-ui)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-714] A prediction_function returning an int isn't validated and passes the upload
Root cause: Validation Gap

A simple model like:

```
def model(df):
return 5
```

passes the upload validation and causes inspector (both prediction and explanation) to break

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-714](https://linear.app/giskard/issue/GSK-714/a-prediction-function-returning-an-int-isnt-validated-and-passes-the)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-751] Feature/ai test v2 test suite execution detailed results
Root cause: Unimplemented Feature Gap



<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-751](https://linear.app/giskard/issue/GSK-751/featureai-test-v2-test-suite-execution-detailed-results)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-644] Huge spaces in the dataset preview table when the data is large
Root cause: Algorithmic Error

[GSK-88](https://giskard.youtrack.cloud/issue/GSK-88)

![](https://uploads.linear.app/5cb5c438-2552-41c0-89ac-b08d19e80d42/e4dbc232-fa42-4320-ba4b-5f0da0869be2/7a3c6ce4-266f-4c69-9f66-16b7453441fe)

Project: Giskard
Reporter: aavtomonov
Created: 2022-05-04

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-644](https://linear.app/giskard/issue/GSK-644/huge-spaces-in-the-dataset-preview-table-when-the-data-is-large)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-680] Multi ML Worker support
Root cause: Unimplemented Feature Gap

## 🚀 Feature Request

Support multiple ML Worker connected at the same time

## 🔈 Motivation

To execute models trained using different python versions and different environments. To let different data scientists work with Giskard at the same time

## 🛰 Alternatives

Disconnecting an actively connected ML Worker and connecting a new one. This is a current solution but it's not great because it only allows a single active worker, so no way to run models trained in different environments at the same time.

## 📎 Additional context

In the settings admin should see all actively connected ML Workers
an active ML Worker should be assigned at a project level

## ❓Questions

Should workers have unique ids to detect if a worker that was selected as "active" for a given project isn't currently available?

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-680](https://linear.app/giskard/issue/GSK-680/multi-ml-worker-support)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-701] Adapt "How to upload models to Giskard" write for community article to the new API



<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-701](https://linear.app/giskard/issue/GSK-701/adapt-how-to-upload-models-to-giskard-write-for-community-article-to)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-546] AI Test v.2
GitHub labels: yt

p:normal

needs specs

[GSK-339](https://giskard.youtrack.cloud/issue/GSK-339)

Project: Giskard
Reporter: aavtomonov
Created: 2022-11-07

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-546](https://linear.app/giskard/issue/GSK-546/ai-test-v2)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-537] Avoid cloning the repo to run GIskard
Root cause: Unimplemented Feature Gap

[GSK-369](https://giskard.youtrack.cloud/issue/GSK-369)

The downsides of the current installation process:

* [ ] the necessity to clone the whole repo, not just docker-compose.yml
* [ ] while cloning the repo, users get the `main` version, but `latest` image tag points to the latest release git tag. There's a chance of mismatch between images and docker-compose.yml
* [ ] implement `giskard backend start` to run `docker compose up -d`

Project: Giskard
Reporter: aavtomonov
Created: 2022-11-16

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-537](https://linear.app/giskard/issue/GSK-537/avoid-cloning-the-repo-to-run-giskard)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-654] Change the default database name from "app" to "giskard"
Root cause: Configuration Error

[GSK-15](https://giskard.youtrack.cloud/issue/GSK-15)

"app" database name can cause conflicts with existing databases if Giskard is configured to use and external DB

Project: Giskard
Reporter: aavtomonov
Created: 2022-04-04

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-654](https://linear.app/giskard/issue/GSK-654/change-the-default-database-name-from-app-to-giskard)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-682] Data labeling
Root cause: Unimplemented Feature Gap

make it possible to change a sample label while inspecting it

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-682](https://linear.app/giskard/issue/GSK-682/data-labeling)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-662] Create Helm chart configuration
Root cause: Unimplemented Feature Gap

[SS-10](https://giskard.youtrack.cloud/issue/SS-10)

Helm chart that uses K8S yaml configs

Project: SaaS
Reporter: aavtomonov
Created: 2022-01-20

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-662](https://linear.app/giskard/issue/GSK-662/create-helm-chart-configuration)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-647] Integrate J2TS and generate DTO interfaces
GitHub labels: yt

[GSK-84](https://giskard.youtrack.cloud/issue/GSK-84)

Project: Giskard
Reporter: aavtomonov
Created: 2022-04-28

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-647](https://linear.app/giskard/issue/GSK-647/integrate-j2ts-and-generate-dto-interfaces)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-610] Installer CLI
Root cause: Unimplemented Feature Gap

[GSK-186](https://giskard.youtrack.cloud/issue/GSK-186)

Project: Giskard
Reporter: jmjm
Created: 2022-07-05

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-610](https://linear.app/giskard/issue/GSK-610/installer-cli)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-783] [GSK-782] ttt



<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-782](https://linear.app/giskard/issue/GSK-782/ttt)</sub>

<sub>[GSK-783](https://linear.app/giskard/issue/GSK-783/[gsk-782]-ttt)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-788] [GSK-786] test linear 2259



<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-786](https://linear.app/giskard/issue/GSK-786/test-linear-2259)</sub>

<sub>[GSK-788](https://linear.app/giskard/issue/GSK-788/[gsk-786]-test-linear-2259)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-787] [GSK-786] test linear 2259



<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-786](https://linear.app/giskard/issue/GSK-786/test-linear-2259)</sub>

<sub>[GSK-787](https://linear.app/giskard/issue/GSK-787/[gsk-786]-test-linear-2259)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-804] [GSK-802] Test another user



<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-802](https://linear.app/giskard/issue/GSK-802/test-another-user)</sub>

<sub>[GSK-804](https://linear.app/giskard/issue/GSK-804/[gsk-802]-test-another-user)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-811] [GSK-810] test another



<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-810](https://linear.app/giskard/issue/GSK-810/test-another)</sub>

<sub>[GSK-811](https://linear.app/giskard/issue/GSK-811/[gsk-810]-test-another)</sub>