Repository: Giskard-AI/giskard-oss
Issue: [GSK-882] Incorrect test counts
Root cause: Algorithmic Error

### Issue Type

Bug

### Source

docker

### Giskard Library Version

1.9.1

### Giskard Server Version

1.5.0

### OS Platform and Distribution

Ubuntu 16.04 LTS

### Python version

3.8.10

### Installed python packages

```shell
aiofiles==22.1.0
aiohttp==3.8.4
aiosignal==1.3.1
aiosqlite==0.18.0
altair==4.2.0
anyio==3.6.2
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arrow==1.2.3
async-timeout==4.0.2
attrs==19.3.0
autocommand==2.2.2
Automat==0.8.0
Babel==2.12.1
backcall==0.2.0
backports.csv==1.0.7
backports.zoneinfo==0.2.1
beautifulsoup4==4.11.2
black==23.1.0
black-nb==0.7
bleach==6.0.0
blinker==1.4
blis==0.7.9
catalogue==2.0.8
certifi==2019.11.28
cffi==1.15.1
chardet==3.0.4
charset-normalizer==3.1.0
checklist==0.0.11
cheroot==9.0.0
CherryPy==18.8.0
click==8.1.3
cloud-init==22.2
cloudpickle==2.2.1
cmake==3.26.0
colorama==0.4.3
comm==0.1.2
command-not-found==0.3
confection==0.0.4
configobj==5.0.6
constantly==15.1.0
cryptography==39.0.2
cymem==2.0.7
datasets==2.10.1
dbus-python==1.2.16
debugpy==1.6.6
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.6
distro==1.4.0
distro-info==0.23ubuntu1
docutils==0.19
eli5==0.13.0
en-core-web-sm==3.5.0
entrypoints==0.3
fastjsonschema==2.16.3
feedparser==6.0.10
filelock==3.10.0
fqdn==1.5.1
frozenlist==1.3.3
fsspec==2023.3.0
future==0.18.3
giskard==1.9.1
googleapis-common-protos==1.58.0
graphviz==0.20.1
great-expectations==0.16.1
grpcio==1.51.1
grpcio-status==1.48.2
httplib2==0.14.0
huggingface-hub==0.13.2
hyperlink==19.0.0
idna==2.8
importlib-metadata==4.13.0
importlib-resources==5.12.0
incremental==16.10.1
inflect==6.0.2
ipykernel==6.21.3
ipython==7.34.0
ipython-genutils==0.2.0
ipywidgets==8.0.4
iso-639==0.4.5
isoduration==20.11.0
jaraco.classes==3.2.3
jaraco.collections==3.8.0
jaraco.context==4.3.0
jaraco.functools==3.6.0
jaraco.text==3.11.1
jedi==0.18.2
Jinja2==3.1.2
joblib==1.2.0
json5==0.9.11
jsonpatch==1.22
jsonpointer==2.0
jsonschema==4.17.3
jupyter==1.0.0
jupyter-client==8.0.3
jupyter-console==6.6.3
jupyter-core==5.3.0
jupyter-events==0.6.3
jupyter-server==2.5.0
jupyter-server-fileid==0.8.0
jupyter-server-terminals==0.4.4
jupyter-server-ydoc==0.6.1
jupyter-ydoc==0.2.3
jupyterlab==3.6.1
jupyterlab-pygments==0.2.2
jupyterlab-server==2.20.0
jupyterlab-widgets==3.0.5
keyring==18.0.1
langcodes==3.3.0
language-selector==0.1
launchpadlib==1.10.13
lazr.restfulclient==0.14.2
lazr.uri==1.0.3
lit==15.0.7
llvmlite==0.39.1
lockfile==0.12.2
lxml==4.9.2
makefun==1.15.1
MarkupSafe==2.1.2
marshmallow==3.19.0
matplotlib-inline==0.1.6
mistune==2.0.5
mixpanel==4.10.0
more-itertools==4.2.0
mpmath==1.3.0
multidict==6.0.4
multiprocess==0.70.14
munch==2.5.0
murmurhash==1.0.9
mypy-extensions==1.0.0
nbclassic==0.5.3
nbclient==0.7.2
nbconvert==7.2.10
nbformat==5.7.3
nest-asyncio==1.5.6
netifaces==0.10.4
networkx==3.0
nltk==3.8.1
notebook==6.5.3
notebook-shim==0.2.2
numba==0.56.4
numpy==1.21.6
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-cupti-cu11==11.7.101
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
nvidia-cufft-cu11==10.9.0.58
nvidia-curand-cu11==10.2.10.91
nvidia-cusolver-cu11==11.4.0.1
nvidia-cusparse-cu11==11.7.4.91
nvidia-nccl-cu11==2.14.3
nvidia-nvtx-cu11==11.7.91
oauthlib==3.2.2
packaging==23.0
pandas==1.5.3
pandocfilters==1.5.0
parso==0.8.3
pathspec==0.11.1
pathy==0.10.1
patternfork-nosql==3.6
pdfminer.six==20221105
pexpect==4.8.0
pickleshare==0.7.5
pip==20.0.2
pkgutil-resolve-name==1.3.10
platformdirs==3.1.1
portalocker==2.7.0
portend==3.1.0
preshed==3.0.8
prometheus-client==0.16.0
prompt-toolkit==3.0.38
protobuf==3.20.3
psutil==5.9.4
ptyprocess==0.7.0
pyarrow==11.0.0
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.21
pydantic==1.10.6
Pygments==2.14.0
PyGObject==3.36.0
PyHamcrest==1.9.0
PyJWT==1.7.1
pymacaroons==0.13.0
PyNaCl==1.3.0
pyOpenSSL==23.0.0
pyparsing==3.0.9
pyrsistent==0.15.5
pyserial==3.4
python-apt==2.0.0+ubuntu0.20.4.7
python-daemon==2.3.2
python-dateutil==2.8.2
python-debian==0.1.36ubuntu1
python-docx==0.8.11
python-json-logger==2.0.7
pytz==2022.7.1
pytz-deprecation-shim==0.1.0.post0
PyYAML==5.3.1
pyzmq==25.0.1
qtconsole==5.4.1
QtPy==2.3.0
regex==2022.10.31
requests==2.28.2
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
requests-unixsocket==0.2.0
responses==0.18.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
ruamel.yaml==0.17.17
ruamel.yaml.clib==0.2.7
scikit-learn==1.0.2
scipy==1.7.3
SecretStorage==2.3.1
Send2Trash==1.8.0
service-identity==18.1.0
setuptools==67.6.0
sgmllib3k==1.0.0
shap==0.41.0
simplejson==3.16.0
six==1.14.0
slicer==0.0.7
smart-open==6.3.0
sniffio==1.3.0
soupsieve==2.4
spacy==3.5.1
spacy-legacy==3.0.12
spacy-loggers==1.0.4
srsly==2.4.6
ssh-import-id==5.10
sympy==1.11.1
systemd-python==234
tabulate==0.9.0
tempora==5.2.1
tenacity==8.2.2
terminado==0.17.1
textblob==0.17.1
thinc==8.1.9
threadpoolctl==3.1.0
tinycss2==1.2.1
tokenize-rt==5.0.0
tokenizers==0.13.2
tomli==2.0.1
toolz==0.12.0
torch==2.0.0
torchdata==0.6.0
torchtext==0.15.1
tornado==6.2
wcwidth==0.2.6
webcolors==1.12
webencodings==0.5.1
websocket-client==1.5.1
wheel==0.34.2
widgetsnbextension==4.0.5
xxhash==3.2.0
y-py==0.5.9
yarl==1.8.2
ypy-websocket==0.8.4
zc.lockfile==3.0.post1
zipp==3.15.0
zope.interface==4.7.1
zstandard==0.20.0
```


### Current Behaviour?

```shell
I created a test suite and populated it with auto-generated tests. 32 tests were created, and on running them, the counts show as  Total: 32 tests.    Executed: 32.    Passed: 8.    Failed: 2. 
Seeing the test status, I can confirm that only 10 tests have run, of which 8 passed and 2 failed. So the correct execution count should be 10.

Also, the "Not executed" filter does not return any tests although I can see a lot of them with the "!" status (I don't know if this means not executed or error).
```


### Standalone code OR list down the steps to reproduce the issue

```shell
Create a test suite with autogenerated tests and run all. Wait for tests to complete.
```


### Relevant log output

_No response_

<sub>[GSK-882](https://linear.app/giskard/issue/GSK-882/incorrect-test-counts)</sub>

Fixed with UI refactoring

<!-- From kevin on Linear -->