Repository: confident-ai/deepeval
Issue: Running `evaluate` within a Google Colab causes `RecursionError`
Root cause: Resource Mishandling

When running `evaluate()` within Google Colab, I get a RecursionError. It happens regardless of using the standalone imported `evaluate` or `dataset.evaluate()`. it doesn't happen when calling `insight_metric.measure()`. I'm using Python 3.10 and Deepeval 0.20.33.

See the example notebook [Deepeval RecursionError.ipynb](https://colab.research.google.com/drive/1aBi8FLyo4qv5Z1m9NYRDjSaxASvR1wY8?usp=sharing). And screenshot:

![image](https://github.com/confident-ai/deepeval/assets/116038/5a6f5099-bdf6-4639-8632-90abf8711cca)


## Snippet that reproduces the bug (same as in Colab)

```python
from deepeval import evaluate
from deepeval.metrics import HallucinationMetric
from deepeval.test_case import LLMTestCase

context=["A man with blond-hair, and a brown shirt drinking out of a public water fountain."]
actual_output="A blond drinking water in public."

test_case = LLMTestCase(
    input="What was the blond doing?",
    actual_output=actual_output,
    context=context
)
metric = HallucinationMetric(minimum_score=0.5)

evaluate([test_case], [metric])
```

<details>
  <summary>Full output</summary>

```
======================================================================

Metrics Summary

  - ✅ Hallucination (score: 0.9959943294525146, minimum_score: 0.5)

For test case:

  - input: What was the blond doing?

  - actual output: A blond drinking water in public.

  - expected output: None

  - context: ['A man with blond-hair, and a brown shirt drinking out of a public water fountain.']

  - retrieval context: None

----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------

----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
Exception in thread Thread-15:
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
----------------------------------------------------------------------
Exception in threading.excepthook:

---------------------------------------------------------------------------

RecursionError                            Traceback (most recent call last)

[/usr/local/lib/python3.10/dist-packages/rich/console.py](https://localhost:8080/#) in print(self, sep, end, style, justify, overflow, no_wrap, emoji, markup, highlight, width, height, crop, soft_wrap, new_line_start, *objects)
   1673         with self:
-> 1674             renderables = self._collect_renderables(
   1675                 objects,

18 frames

[/usr/local/lib/python3.10/dist-packages/rich/console.py](https://localhost:8080/#) in _collect_renderables(self, objects, sep, end, justify, emoji, markup, highlight)
   1537                 append_text(
-> 1538                     self.render_str(
   1539                         renderable, emoji=emoji, markup=markup, highlighter=_highlighter

[/usr/local/lib/python3.10/dist-packages/rich/console.py](https://localhost:8080/#) in render_str(self, text, style, justify, overflow, emoji, markup, highlight, highlighter)
   1449         if _highlighter is not None:
-> 1450             highlight_text = _highlighter(str(rich_text))
   1451             highlight_text.copy_styles(rich_text)

[/usr/local/lib/python3.10/dist-packages/rich/highlighter.py](https://localhost:8080/#) in __call__(self, text)
     32         if isinstance(text, str):
---> 33             highlight_text = Text(text)
     34         elif isinstance(text, Text):

[/usr/local/lib/python3.10/dist-packages/rich/text.py](https://localhost:8080/#) in __init__(self, text, style, justify, overflow, no_wrap, end, tab_size, spans)
    154     ) -> None:
--> 155         sanitized_text = strip_control_codes(text)
    156         self._text = [sanitized_text]

[/usr/local/lib/python3.10/dist-packages/rich/control.py](https://localhost:8080/#) in strip_control_codes(text, _translate_table)
    197     """
--> 198     return text.translate(_translate_table)
    199 

RecursionError: maximum recursion depth exceeded while calling a Python object


During handling of the above exception, another exception occurred:

RecursionError                            Traceback (most recent call last)

[<ipython-input-8-d4f13febbae1>](https://localhost:8080/#) in <cell line: 15>()
     13 metric = HallucinationMetric(minimum_score=0.5)
     14 
---> 15 evaluate([test_case], [metric])

[/usr/local/lib/python3.10/dist-packages/deepeval/evaluate.py](https://localhost:8080/#) in evaluate(test_cases, metrics)
    115         for test_result in test_results:
    116             print_test_result(test_result)
--> 117         print("\n" + "-" * 70)
    118 
    119         test_run_manager.wrap_up_test_run(display_table=False)

[/usr/local/lib/python3.10/dist-packages/rich/file_proxy.py](https://localhost:8080/#) in write(self, text)
     41         if lines:
     42             console = self.__console
---> 43             with console:
     44                 output = Text("\n").join(
     45                     self.__ansi_decoder.decode_line(line) for line in lines

[/usr/local/lib/python3.10/dist-packages/rich/console.py](https://localhost:8080/#) in __exit__(self, exc_type, exc_value, traceback)
    863     def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
    864         """Exit buffer context."""
--> 865         self._exit_buffer()
    866 
    867     def begin_capture(self) -> None:

[/usr/local/lib/python3.10/dist-packages/rich/console.py](https://localhost:8080/#) in _exit_buffer(self)
    821         """Leave buffer context, and render content if required."""
    822         self._buffer_index -= 1
--> 823         self._check_buffer()
    824 
    825     def set_live(self, live: "Live") -> None:

[/usr/local/lib/python3.10/dist-packages/rich/console.py](https://localhost:8080/#) in _check_buffer(self)
   2005                     from .jupyter import display
   2006 
-> 2007                     display(self._buffer, self._render_buffer(self._buffer[:]))
   2008                     del self._buffer[:]
   2009                 else:

[/usr/local/lib/python3.10/dist-packages/rich/jupyter.py](https://localhost:8080/#) in display(segments, text)
     89         from IPython.display import display as ipython_display
     90 
---> 91         ipython_display(jupyter_renderable)
     92     except ModuleNotFoundError:
     93         # Handle the case where the Console has force_jupyter=True,

[/usr/local/lib/python3.10/dist-packages/IPython/core/display.py](https://localhost:8080/#) in display(include, exclude, metadata, transient, display_id, *objs, **kwargs)
    325                 # kwarg-specified metadata gets precedence
    326                 _merge(md_dict, metadata)
--> 327             publish_display_data(data=format_dict, metadata=md_dict, **kwargs)
    328     if display_id:
    329         return DisplayHandle(display_id)

[/usr/local/lib/python3.10/dist-packages/IPython/core/display.py](https://localhost:8080/#) in publish_display_data(data, metadata, source, transient, **kwargs)
    117         kwargs['transient'] = transient
    118 
--> 119     display_pub.publish(
    120         data=data,
    121         metadata=metadata,

[/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py](https://localhost:8080/#) in publish(self, data, metadata, source, transient, update)
    113             If True, send an update_display_data message instead of display_data.
    114         """
--> 115         self._flush_streams()
    116         if metadata is None:
    117             metadata = {}

[/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py](https://localhost:8080/#) in _flush_streams(self)
     80     def _flush_streams(self):
     81         """flush IO Streams prior to display"""
---> 82         sys.stdout.flush()
     83         sys.stderr.flush()
     84 

[/usr/local/lib/python3.10/dist-packages/rich/file_proxy.py](https://localhost:8080/#) in flush(self)
     51         output = "".join(self.__buffer)
     52         if output:
---> 53             self.__console.print(output)
     54         del self.__buffer[:]
     55 

[/usr/local/lib/python3.10/dist-packages/rich/console.py](https://localhost:8080/#) in print(self, sep, end, style, justify, overflow, no_wrap, emoji, markup, highlight, width, height, crop, soft_wrap, new_line_start, *objects)
   1671             crop = False
   1672         render_hooks = self._render_hooks[:]
-> 1673         with self:
   1674             renderables = self._collect_renderables(
   1675                 objects,

... last 10 frames repeated, from the frame below ...

[/usr/local/lib/python3.10/dist-packages/rich/console.py](https://localhost:8080/#) in __exit__(self, exc_type, exc_value, traceback)
    863     def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
    864         """Exit buffer context."""
--> 865         self._exit_buffer()
    866 
    867     def begin_capture(self) -> None:

RecursionError: maximum recursion depth exceeded
```
</details>

Edit: Hid the full output behind a collapsible area for visibility.

@manuelmorales can you show me your environment? Either `pip freeze` if you're on pip or `poetry show` if you're using poetry.

Sure @penguine-ip , here it is:
<details>
<summary><code>pip freeze</code> output.</summary>
<pre>
absl-py==1.4.0
aiohttp==3.9.1
aiosignal==1.3.1
alabaster==0.7.13
albumentations==1.3.1
altair==4.2.2
anyio==3.7.1
appdirs==1.4.4
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
array-record==0.5.0
arviz==0.15.1
astropy==5.3.4
astunparse==1.6.3
async-timeout==4.0.3
atpublic==4.0
attrs==23.1.0
audioread==3.0.1
autograd==1.6.2
Babel==2.13.1
backcall==0.2.0
beautifulsoup4==4.11.2
bidict==0.22.1
bigframes==0.15.0
bleach==6.1.0
blinker==1.4
blis==0.7.11
blosc2==2.0.0
bokeh==3.3.2
bqplot==0.12.42
branca==0.7.0
build==1.0.3
CacheControl==0.13.1
cachetools==5.3.2
catalogue==2.0.10
certifi==2023.11.17
cffi==1.16.0
chardet==5.2.0
charset-normalizer==3.3.2
chex==0.1.7
click==8.1.7
click-plugins==1.1.1
cligj==0.7.2
cloudpickle==2.2.1
cmake==3.27.9
cmdstanpy==1.2.0
colorcet==3.0.1
colorlover==0.3.0
colour==0.1.5
community==1.0.0b1
confection==0.1.4
cons==0.4.6
contextlib2==21.6.0
contourpy==1.2.0
cryptography==41.0.7
cufflinks==0.17.3
cupy-cuda11x==11.0.0
cvxopt==1.3.2
cvxpy==1.3.2
cycler==0.12.1
cymem==2.0.8
Cython==3.0.6
dask==2023.8.1
dataclasses-json==0.6.3
datascience==0.17.6
datasets==2.15.0
db-dtypes==1.1.1
dbus-python==1.2.18
debugpy==1.6.6
decorator==4.4.2
deepeval==0.20.35
defusedxml==0.7.1
detoxify==0.5.1
dill==0.3.7
diskcache==5.6.3
distributed==2023.8.1
distro==1.7.0
dlib==19.24.2
dm-tree==0.1.8
docutils==0.18.1
dopamine-rl==4.0.6
duckdb==0.9.2
earthengine-api==0.1.381
easydict==1.11
ecos==2.0.12
editdistance==0.6.2
eerepr==0.0.4
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.6.0/en_core_web_sm-3.6.0-py3-none-any.whl#sha256=83276fc78a70045627144786b52e1f2728ad5e29e5e43916ec37ea9c26a11212
entrypoints==0.4
et-xmlfile==1.1.0
etils==1.5.2
etuples==0.3.9
exceptiongroup==1.2.0
execnet==2.0.2
fastai==2.7.13
fastcore==1.5.29
fastdownload==0.0.7
fastjsonschema==2.19.0
fastprogress==1.0.3
fastrlock==0.8.2
filelock==3.13.1
fiona==1.9.5
firebase-admin==5.3.0
Flask==2.2.5
flatbuffers==23.5.26
flax==0.7.5
folium==0.14.0
fonttools==4.46.0
frozendict==2.3.10
frozenlist==1.4.0
fsspec==2023.6.0
future==0.18.3
gast==0.5.4
gcsfs==2023.6.0
GDAL==3.4.3
gdown==4.6.6
geemap==0.29.6
gensim==4.3.2
geocoder==1.38.1
geographiclib==2.0
geopandas==0.13.2
geopy==2.3.0
gin-config==0.5.0
glob2==0.7
google==2.0.3
google-ai-generativelanguage==0.3.3
google-api-core==2.11.1
google-api-python-client==2.84.0
google-auth==2.17.3
google-auth-httplib2==0.1.1
google-auth-oauthlib==1.0.0
google-cloud-aiplatform==1.36.4
google-cloud-bigquery==3.12.0
google-cloud-bigquery-connection==1.12.1
google-cloud-bigquery-storage==2.23.0
google-cloud-core==2.3.3
google-cloud-datastore==2.15.2
google-cloud-firestore==2.11.1
google-cloud-functions==1.13.3
google-cloud-iam==2.12.2
google-cloud-language==2.9.1
google-cloud-resource-manager==1.10.4
google-cloud-storage==2.8.0
google-cloud-translate==3.11.3
google-colab @ file:///colabtools/dist/google-colab-1.0.0.tar.gz#sha256=d402ff04028f211431c19a5a99d65e12eecf2eb3911ce18e39c83b276e17afcc
google-crc32c==1.5.0
google-generativeai==0.2.2
google-pasta==0.2.0
google-resumable-media==2.6.0
googleapis-common-protos==1.61.0
googledrivedownloader==0.4
graphviz==0.20.1
greenlet==3.0.1
grpc-google-iam-v1==0.12.7
grpcio==1.59.3
grpcio-status==1.48.2
gspread==3.4.2
gspread-dataframe==3.3.1
gym==0.25.2
gym-notices==0.0.8
h11==0.14.0
h5netcdf==1.3.0
h5py==3.9.0
holidays==0.38
holoviews==1.17.1
html5lib==1.1
httpcore==1.0.2
httpimport==1.3.1
httplib2==0.22.0
httpx==0.25.2
huggingface-hub==0.19.4
humanize==4.7.0
hyperopt==0.2.7
ibis-framework==6.2.0
idna==3.6
imageio==2.31.6
imageio-ffmpeg==0.4.9
imagesize==1.4.1
imbalanced-learn==0.10.1
imgaug==0.4.0
importlib-metadata==7.0.0
importlib-resources==6.1.1
imutils==0.5.4
inflect==7.0.0
iniconfig==2.0.0
install==1.3.5
intel-openmp==2023.2.0
ipyevents==2.0.2
ipyfilechooser==0.6.0
ipykernel==5.5.6
ipyleaflet==0.18.0
ipython==7.34.0
ipython-genutils==0.2.0
ipython-sql==0.5.0
ipytree==0.2.2
ipywidgets==7.7.1
itsdangerous==2.1.2
jax==0.4.20
jaxlib @ https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.4.20+cuda11.cudnn86-cp310-cp310-manylinux2014_x86_64.whl#sha256=01be66238133f884bf5adf15cd7eaaf8445f9d4b056c5c64df28a997a6aff2fe
jeepney==0.7.1
jieba==0.42.1
Jinja2==3.1.2
joblib==1.3.2
jsonpatch==1.33
jsonpickle==3.0.2
jsonpointer==2.4
jsonschema==4.19.2
jsonschema-specifications==2023.11.2
jupyter-client==6.1.12
jupyter-console==6.1.0
jupyter-server==1.24.0
jupyter_core==5.5.0
jupyterlab-widgets==3.0.9
jupyterlab_pygments==0.3.0
kaggle==1.5.16
keras==2.14.0
keyring==23.5.0
kiwisolver==1.4.5
langchain==0.0.350
langchain-community==0.0.3
langchain-core==0.1.0
langcodes==3.3.0
langsmith==0.0.70
launchpadlib==1.10.16
lazr.restfulclient==0.14.4
lazr.uri==1.0.6
lazy_loader==0.3
libclang==16.0.6
librosa==0.10.1
lida==0.0.10
lightgbm==4.1.0
linkify-it-py==2.0.2
llmx==0.0.15a0
llvmlite==0.41.1
locket==1.0.0
logical-unification==0.4.6
lxml==4.9.3
malloy==2023.1067
Markdown==3.5.1
markdown-it-py==3.0.0
MarkupSafe==2.1.3
marshmallow==3.20.1
matplotlib==3.7.1
matplotlib-inline==0.1.6
matplotlib-venn==0.11.9
mdit-py-plugins==0.4.0
mdurl==0.1.2
miniKanren==1.0.3
missingno==0.5.2
mistune==0.8.4
mizani==0.9.3
mkl==2023.2.0
ml-dtypes==0.2.0
mlxtend==0.22.0
more-itertools==10.1.0
moviepy==1.0.3
mpmath==1.3.0
msgpack==1.0.7
multidict==6.0.4
multipledispatch==1.0.0
multiprocess==0.70.15
multitasking==0.0.11
murmurhash==1.0.10
music21==9.1.0
mypy-extensions==1.0.0
natsort==8.4.0
nbclassic==1.0.0
nbclient==0.9.0
nbconvert==6.5.4
nbformat==5.9.2
nest-asyncio==1.5.8
networkx==3.2.1
nibabel==4.0.2
nltk==3.8.1
notebook==6.5.5
notebook_shim==0.2.3
numba==0.58.1
numexpr==2.8.7
numpy==1.23.5
oauth2client==4.1.3
oauthlib==3.2.2
openai==1.3.9
opencv-contrib-python==4.8.0.76
opencv-python==4.8.0.76
opencv-python-headless==4.8.1.78
openpyxl==3.1.2
opt-einsum==3.3.0
optax==0.1.7
orbax-checkpoint==0.4.4
osqp==0.6.2.post8
packaging==23.2
pandas==1.5.3
pandas-datareader==0.10.0
pandas-gbq==0.19.2
pandas-stubs==1.5.3.230304
pandocfilters==1.5.0
panel==1.3.4
param==2.0.1
parso==0.8.3
parsy==2.1
partd==1.4.1
pathlib==1.0.1
pathy==0.10.3
patsy==0.5.4
peewee==3.17.0
pexpect==4.9.0
pickleshare==0.7.5
Pillow==9.4.0
pip-tools==6.13.0
platformdirs==4.1.0
plotly==5.15.0
plotnine==0.12.4
pluggy==1.3.0
polars==0.17.3
pooch==1.8.0
portalocker==2.8.2
portpicker==1.5.2
prefetch-generator==1.0.3
preshed==3.0.9
prettytable==3.9.0
proglog==0.1.10
progressbar2==4.2.0
prometheus-client==0.19.0
promise==2.3
prompt-toolkit==3.0.41
prophet==1.1.5
proto-plus==1.22.3
protobuf==4.21.6
psutil==5.9.5
psycopg2==2.9.9
ptyprocess==0.7.0
py-cpuinfo==9.0.0
py4j==0.10.9.7
pyarrow==10.0.1
pyarrow-hotfix==0.6
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycocotools==2.0.7
pycparser==2.21
pyct==0.5.0
pydantic==1.10.13
pydata-google-auth==1.8.2
pydot==1.4.2
pydot-ng==2.0.0
pydotplus==2.0.2
PyDrive==1.3.1
PyDrive2==1.6.3
pyerfa==2.0.1.1
pygame==2.5.2
Pygments==2.16.1
PyGObject==3.42.1
PyJWT==2.3.0
pymc==5.7.2
pymystem3==0.2.0
PyOpenGL==3.1.7
pyOpenSSL==23.3.0
pyparsing==3.1.1
pyperclip==1.8.2
pyproj==3.6.1
pyproject_hooks==1.0.0
pysbd==0.3.4
pyshp==2.3.1
PySocks==1.7.1
pytensor==2.14.2
pytest==7.4.3
pytest-xdist==3.5.0
python-apt==0.0.0
python-box==7.1.1
python-dateutil==2.8.2
python-louvain==0.16
python-slugify==8.0.1
python-utils==3.8.1
pytz==2023.3.post1
pyviz_comms==3.0.0
PyWavelets==1.5.0
PyYAML==6.0.1
pyzmq==23.2.1
qdldl==0.1.7.post0
qudida==0.0.4
ragas==0.0.22
ratelim==0.1.6
referencing==0.31.1
regex==2023.6.3
requests==2.31.0
requests-oauthlib==1.3.1
requirements-parser==0.5.0
rich==13.7.0
rouge-score==0.1.2
rpds-py==0.13.2
rpy2==3.4.2
rsa==4.9
safetensors==0.4.1
scikit-image==0.19.3
scikit-learn==1.2.2
scipy==1.11.4
scooby==0.9.2
scs==3.2.4.post1
seaborn==0.12.2
SecretStorage==3.3.1
Send2Trash==1.8.2
sentence-transformers==2.2.2
sentencepiece==0.1.99
sentry-sdk==1.39.1
shapely==2.0.2
six==1.16.0
sklearn-pandas==2.2.0
smart-open==6.4.0
sniffio==1.3.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
soundfile==0.12.1
soupsieve==2.5
soxr==0.3.7
spacy==3.6.1
spacy-legacy==3.0.12
spacy-loggers==1.0.5
Sphinx==5.0.2
sphinxcontrib-applehelp==1.0.7
sphinxcontrib-devhelp==1.0.5
sphinxcontrib-htmlhelp==2.0.4
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.6
sphinxcontrib-serializinghtml==1.1.9
SQLAlchemy==2.0.23
sqlglot==17.16.2
sqlparse==0.4.4
srsly==2.4.8
stanio==0.3.0
statsmodels==0.14.0
sympy==1.12
tables==3.8.0
tabulate==0.9.0
tbb==2021.11.0
tblib==3.0.0
tenacity==8.2.3
tensorboard==2.14.1
tensorboard-data-server==0.7.2
tensorflow==2.14.0
tensorflow-datasets==4.9.3
tensorflow-estimator==2.14.0
tensorflow-gcs-config==2.14.0
tensorflow-hub==0.15.0
tensorflow-io-gcs-filesystem==0.34.0
tensorflow-metadata==1.14.0
tensorflow-probability==0.22.0
tensorstore==0.1.45
termcolor==2.4.0
terminado==0.18.0
text-unidecode==1.3
textblob==0.17.1
tf-slim==1.1.0
thinc==8.1.12
threadpoolctl==3.2.0
tifffile==2023.9.26
tiktoken==0.5.2
tinycss2==1.2.1
tokenizers==0.12.1
toml==0.10.2
tomli==2.0.1
toolz==0.12.0
torch @ https://download.pytorch.org/whl/cu118/torch-2.1.0%2Bcu118-cp310-cp310-linux_x86_64.whl#sha256=a81b554184492005543ddc32e96469f9369d778dedd195d73bda9bed407d6589
torchaudio @ https://download.pytorch.org/whl/cu118/torchaudio-2.1.0%2Bcu118-cp310-cp310-linux_x86_64.whl#sha256=cdfd0a129406155eee595f408cafbb92589652da4090d1d2040f5453d4cae71f
torchdata==0.7.0
torchsummary==1.5.1
torchtext==0.16.0
torchvision @ https://download.pytorch.org/whl/cu118/torchvision-0.16.0%2Bcu118-cp310-cp310-linux_x86_64.whl#sha256=033712f65d45afe806676c4129dfe601ad1321d9e092df62b15847c02d4061dc
tornado==6.3.2
tqdm==4.66.1
traitlets==5.7.1
traittypes==0.2.1
transformers==4.22.1
triton==2.1.0
tweepy==4.14.0
typer==0.9.0
types-pytz==2023.3.1.1
types-setuptools==69.0.0.0
typing-inspect==0.9.0
typing_extensions==4.5.0
tzlocal==5.2
uc-micro-py==1.0.2
uritemplate==4.1.1
urllib3==2.0.7
vega-datasets==0.9.0
wadllib==1.3.6
wasabi==1.1.2
wcwidth==0.2.12
webcolors==1.13
webencodings==0.5.1
websocket-client==1.7.0
Werkzeug==3.0.1
widgetsnbextension==3.6.6
wordcloud==1.9.2
wrapt==1.14.1
xarray==2023.7.0
xarray-einstats==0.6.0
xgboost==2.0.2
xlrd==2.0.1
xxhash==3.4.1
xyzservices==2023.10.1
yarl==1.9.3
yellowbrick==1.5
yfinance==0.2.32
zict==3.0.0
zipp==3.17.0
</pre>
</details>

Thanks for providing this, can you please come to our discord and we can dive deeper there: https://discord.com/invite/a3K9c8GRGt

For now I will try to reproduce the error in a colab.

Sure. I just joined and DM'd you. 

> For now I will try to reproduce the error in a colab.

If it is any useful, I created [this Collab](https://colab.research.google.com/drive/1aBi8FLyo4qv5Z1m9NYRDjSaxASvR1wY8?usp=sharing) to reproduce it. 

I am running into a similar issue running evaluate() on jupyter

I couldn't produce it, i'm running the colab in the quickstart repo: https://colab.research.google.com/drive/1PPxYEBa6eu__LquGoFFJZkhYgWVYE6kh?usp=sharing, what version are you using?

Facing the same issue it does not always happen but I've noticed it in following scenario

```python
answer_relevancy_metric = AnswerRelevancyMetric()
faithfulness_metric = FaithfulnessMetric()
answer="""Indian Railways is a statutory body under the ownership of the Ministry of Railways of the Government of India.

Q: How many operational zones is Indian Railways divided into?
A: Indian Railways is divided into 17 operational zones geographically.

Q: What is the running track length of Indian Railways?
A: The running track length of Indian Railways is 104,647 km (65,025 mi).

Q: When was the first electric train run in India?
A: The first electric train ran in Bombay on DC traction in 1925.

Q: How many employees does Indian Railways have?
A: Indian Railways has more than 1.2 million employees, making it the world's ninth-largest employer and India's second largest employer.

Q: When was Indian Railway Catering and Tourism Corporation (IRCTC) established?
A: Indian Railway Catering and Tourism Corporation (IRCTC) was established on 27 September 1999."""
contexts=["Indian Railways is a statutory body under the ownership of the Ministry of Railways of the Government of India that operates India's national railway system. It is headed by a Railway Board whose chairman reports to the Ministry of Railways. It is organized into separate functional groups or verticals while divided into 17 operational zones geographically. Each zone, headed by a General Manager, is semi-autonomous thus creating a matrix organization where the functional branches are under dual control.", "Indian Railways (IR) is a statutory body under the ownership of the Ministry of Railways of the Government of India that operates India's national railway system. As of 2023, it manages the fourth largest national railway system by size with a running track length of 104,647 km (65,025 mi) and route length of 68,426 km (42,518 mi) of which 60,451 km (37,563 mi) is electrified. With more than 1.2 million employees, it is the world's ninth-largest employer and India's second largest employer.\nThe first steam operated railway operated in 1837 in Madras with the first passenger operating in 1853 between Bombay and Thane. In 1925, the first electric train ran in Bombay on DC traction. The first locomotive manufacturing unit was commissioned in 1950 at Chittaranjan with the first coach manufacturing unit set-up at Madras in 1955. Various companies operating railways across the country were re-organised into six regional zones in 1951, which were gradually expanded to 19 zones.", "The Ministry of Railways is a ministry in the Government of India, responsible for the country's rail transport. Indian Railways is a statutory body managed by the railway board under the ownership of the ministry that operates the national railway system. The ministry along with the railway board is housed inside Rail Bhawan in New Delhi. With more than 1.2 million employees, it is one of the world's largest employers.", "== History ==\nIndian Railway Catering and Tourism Corporation (IRCTC) was established on 27 September 1999, as a public sector undertaking completely owned by the Government of India through the Indian Railways. In May 2008, it was classed as a Miniratna public corporation, which allowed it a certain degree of financial autonomy. The company was listed on the National Stock Exchange in 2019, following which the Government of India's holding was reduced to 87%, with the remaining shares being publicly traded. In December 2020, the Government of India divested another 20%, reducing its holding in the IRCTC to 67%. In December 2022, the government dis-invested further 5% of its share, reducing its ownership to 62.4%.\n\n\n== Services =="]

test_case = LLMTestCase(
  input="Who owns indian Railways?",
  actual_output=answer,
  expected_output="We are a statutory body under the ownership of the Ministry of Railways of the Government of India.",
  retrieval_context=contexts,
)
dataset = EvaluationDataset(test_cases=[test_case])
evaluate(dataset, metrics=[answer_relevancy_metric, faithfulness_metric])
```

Output:
```======================================================================

Metrics Summary

  - ❌ Answer Relevancy (score: 0.16666666666666666, threshold: 0.5, evaluation model: gpt-4-0125-preview, reason: 
The score is 0.17 because the response provided a lot of detailed information about Indian Railways, such as its 
operational zones, track length, history of electric trains, employment statistics, and the establishment of IRCTC,
which, while informative, did not directly address the specific question about the ownership of Indian Railways. 
This resulted in a low relevancy score as the actual question was not directly answered amidst the abundance of 
unrelated facts.)

  - ✅ Faithfulness (score: 1.0, threshold: 0.5, evaluation model: gpt-4-0125-preview, reason: The score is 1.00 
because the actual output is perfectly aligned with the information presented in the retrieval context, showcasing 
a high level of faithfulness without any contradictions. Great job on maintaining accuracy!)

For test case:

  - input: Who owns indian Railways?

  - actual output: Indian Railways is a statutory body under the ownership of the Ministry of Railways of the 
Government of India.

Q: How many operational zones is Indian Railways divided into?
A: Indian Railways is divided into 17 operational zones geographically.

Q: What is the running track length of Indian Railways?
A: The running track length of Indian Railways is 104,647 km (65,025 mi).

Q: When was the first electric train run in India?
A: The first electric train ran in Bombay on DC traction in 1925.

Q: How many employees does Indian Railways have?
A: Indian Railways has more than 1.2 million employees, making it the world's ninth-largest employer and India's 
second largest employer.

Q: When was Indian Railway Catering and Tourism Corporation (IRCTC) established?
A: Indian Railway Catering and Tourism Corporation (IRCTC) was established on 27 September 1999.
A: Indian Railway Catering and Tourism Corporation (IRCTC) was established on 27 September 1999.
A: Indian Railway Catering and Tourism Corporation (IRCTC) was established on 27 September 1999.
A: Indian Railway Catering and Tourism Corporation (IRCTC) was established on 27 September 1999.
A: Indian Railway Catering and Tourism Corporation (IRCTC) was established on 27 September 1999.
A: Indian Railway Catering and Tourism Corporation (IRCTC) was established on 27 September 1999.
A: Indian Railway Catering and Tourism Corporation (IRCTC) was established on 27 September 1999.
A: Indian Railway Catering and Tourism Corporation (IRCTC) was established on 27 September 1999.
IT GOES ON
```

@kubre what version of deepeval is this?

@penguine-ip It should be the latest as I'm running it in colab. Also if you suppress output using %%capture it works fine
```
Collecting deepeval
  Downloading deepeval-0.20.81-py3-none-any.whl (128 kB)
```