Repository: confident-ai/deepeval
Issue: llama index integration broken at latest versions
Root cause: External Dependency Breakage

**Describe the bug**
llama index and deepeval don't work together in latest versions



**To Reproduce**
$ egrep -e '(llama|deep)' requirements.txt 
llama-index==0.9.38
deepeval==0.20.53

Use these instructions: https://docs.llamaindex.ai/en/stable/module_guides/observability/observability.html#deepeval

**Expected behavior**
Basic guide from Llama index should work.

Result:

```
    from deepeval.tracing.integrations.llama_index import LlamaIndexCallbackHandler
ModuleNotFoundError: No module named 'deepeval.tracing.integrations'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/yvesjunqueira/src/github.com/artell-ai/clinica-continuum/main.py", line 7, in <module>
    set_global_handler("deepeval")
  File "/Users/yvesjunqueira/Library/Python/3.9/lib/python/site-packages/llama_index/callbacks/global_handlers.py", line 20, in set_global_handler
    llama_index.global_handler = create_global_handler(eval_mode, **eval_params)
  File "/Users/yvesjunqueira/Library/Python/3.9/lib/python/site-packages/llama_index/callbacks/global_handlers.py", line 36, in create_global_handler
    handler = deepeval_callback_handler(**eval_params)
  File "/Users/yvesjunqueira/Library/Python/3.9/lib/python/site-packages/llama_index/callbacks/deepeval_callback.py", line 10, in deepeval_callback_handler
    raise ImportError("Please install DeepEval with `pip install -U deepeval`")
```

**Additional context**

Here are the files that get installed by the latest version of deepeval in the tracing module:

```
$ find /Users/yvesjunqueira/Library/Python/3.9/lib/python/site-packages/deepeval/tracing
/Users/yvesjunqueira/Library/Python/3.9/lib/python/site-packages/deepeval/tracing
/Users/yvesjunqueira/Library/Python/3.9/lib/python/site-packages/deepeval/tracing/tracing.py
/Users/yvesjunqueira/Library/Python/3.9/lib/python/site-packages/deepeval/tracing/__init__.py
```

Weirdly, there are no `integrations` file or directory, which suggests a problem with the dist scripts?

```
$ find /Users/yvesjunqueira/Library/Python/3.9/lib/python/site-packages/deepeval |grep integr
(empty)
```

A bit more details to help you debug this:

If you download the .tar.gz from pypi.org manually, there is no integration directory:

```
$ tar tzvf deepeval-0.20.53\ \(1\).tar.gz |grep integr
(empty)
```

but if you download it from github, there is:

```
$ tar tzvf deepeval-0.20.53.tar.gz |grep integr
drwxrwxr-x  0 root   root        0 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/
drwxrwxr-x  0 root   root        0 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/harness/
-rw-rw-r--  0 root   root       66 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/harness/__init__.py
-rw-rw-r--  0 root   root      489 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/harness/callback.py
drwxrwxr-x  0 root   root        0 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/hugging_face/
-rw-rw-r--  0 root   root       75 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/hugging_face/__init__.py
-rw-rw-r--  0 root   root     7114 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/hugging_face/callback.py
-rw-rw-r--  0 root   root     3630 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/hugging_face/rich_manager.py
drwxrwxr-x  0 root   root        0 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/hugging_face/tests/
-rw-rw-r--  0 root   root     4657 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/hugging_face/tests/test_callbacks.py
-rw-rw-r--  0 root   root     1746 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/hugging_face/utils.py
drwxrwxr-x  0 root   root        0 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/llama_index/
-rw-rw-r--  0 root   root      500 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/llama_index/__init__.py
-rw-rw-r--  0 root   root     4794 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/llama_index/callback.py
-rw-rw-r--  0 root   root     8163 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/llama_index/evaluators.py
drwxrwxr-x  0 root   root        0 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/llama_index/tests/
-rw-rw-r--  0 root   root      921 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/llama_index/tests/test_evaluators.py
-rw-rw-r--  0 root   root      227 Jan 25 07:09 deepeval-0.20.53/deepeval/integrations/llama_index/utils.py
-rw-rw-r--  0 root   root      663 Jan 25 07:09 deepeval-0.20.53/docs/docs/integrations-introduction.mdx
-rw-rw-r--  0 root   root     7972 Jan 25 07:09 deepeval-0.20.53/docs/docs/integrations-llamaindex.mdx
```

I hope this helps.

The second problem that needs to be fixed is that the path of the library has changed so it should be updated in Llama index's code.

from deepeval.integrations.llama_index import LlamaIndexCallbackHandler
instead of
from deepeval.tracing.integrations.llama_index import LlamaIndexCallbackHandler

I don't think the first path ever worked in the wild so it should be safe to update llama index's code IMHO.

@nictuku thanks for bringing it up, the llamaindex docs aren't up to date but as you pointed out `deepeval.integrations.llama_index import LlamaIndexCallbackHandler` is the new import path, can you try that out and let me know how it goes?

@penguine-ip thanks for the reply, but like I said, that is not the only problem. Your pypi modules are shipping without the integration directory.

```
$ curl -s https://files.pythonhosted.org/packages/09/67/496b02fc288ba2d072223456fb1d7b0ded8a956cc0694502248b31a1792e/deepeval-0.20.53.tar.gz | tar -tzvf -
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/
-rw-r--r--  0 jeffreyip staff   11358 Oct  6 07:22 deepeval-0.20.53/LICENSE.md
-rw-r--r--  0 jeffreyip staff     834 Jan 25 07:10 deepeval-0.20.53/PKG-INFO
-rw-r--r--  0 jeffreyip staff   11274 Jan 25 04:59 deepeval-0.20.53/README.md
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/deepeval/
-rw-r--r--  0 jeffreyip staff    1582 Jan  3 12:54 deepeval-0.20.53/deepeval/__init__.py
-rw-r--r--  0 jeffreyip staff      29 Jan 25 07:09 deepeval-0.20.53/deepeval/_version.py
-rw-r--r--  0 jeffreyip staff   10211 Jan 19 11:13 deepeval-0.20.53/deepeval/api.py
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/deepeval/chat_completion/
-rw-r--r--  0 jeffreyip staff       0 Oct 27 10:26 deepeval-0.20.53/deepeval/chat_completion/__init__.py
-rw-r--r--  0 jeffreyip staff    1309 Dec 28 14:04 deepeval-0.20.53/deepeval/chat_completion/retry.py
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/deepeval/check/
-rw-r--r--  0 jeffreyip staff      25 Dec  6 20:48 deepeval-0.20.53/deepeval/check/__init__.py
-rw-r--r--  0 jeffreyip staff     108 Dec  6 20:48 deepeval-0.20.53/deepeval/check/benchmarks.py
-rw-r--r--  0 jeffreyip staff     402 Dec  6 20:48 deepeval-0.20.53/deepeval/check/check.py
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/deepeval/cli/
-rw-r--r--  0 jeffreyip staff       0 Aug 29 11:44 deepeval-0.20.53/deepeval/cli/__init__.py
-rw-r--r--  0 jeffreyip staff    3865 Jan 20 12:32 deepeval-0.20.53/deepeval/cli/main.py
-rw-r--r--  0 jeffreyip staff    2669 Jan 23 22:59 deepeval-0.20.53/deepeval/cli/test.py
-rw-r--r--  0 jeffreyip staff     248 Dec 20 02:50 deepeval-0.20.53/deepeval/constants.py
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/deepeval/dataset/
-rw-r--r--  0 jeffreyip staff      63 Dec 28 14:04 deepeval-0.20.53/deepeval/dataset/__init__.py
-rw-r--r--  0 jeffreyip staff     702 Jan 23 09:34 deepeval-0.20.53/deepeval/dataset/api.py
-rw-r--r--  0 jeffreyip staff   12606 Jan 23 22:59 deepeval-0.20.53/deepeval/dataset/dataset.py
-rw-r--r--  0 jeffreyip staff     981 Jan 23 09:34 deepeval-0.20.53/deepeval/dataset/utils.py
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/deepeval/decorators/
-rw-r--r--  0 jeffreyip staff       0 Oct 18 08:55 deepeval-0.20.53/deepeval/decorators/__init__.py
-rw-r--r--  0 jeffreyip staff     188 Oct 18 08:55 deepeval-0.20.53/deepeval/decorators/hyperparameters.py
-rw-r--r--  0 jeffreyip staff    5077 Jan 20 02:41 deepeval-0.20.53/deepeval/evaluate.py
-rw-r--r--  0 jeffreyip staff    2223 Jan 19 09:58 deepeval-0.20.53/deepeval/event.py
-rw-r--r--  0 jeffreyip staff    1993 Dec 28 14:04 deepeval-0.20.53/deepeval/key_handler.py
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/deepeval/metrics/
-rw-r--r--  0 jeffreyip staff    1113 Jan 21 12:28 deepeval-0.20.53/deepeval/metrics/__init__.py
-rw-r--r--  0 jeffreyip staff    4218 Jan 19 08:47 deepeval-0.20.53/deepeval/metrics/answer_relevancy.py
-rw-r--r--  0 jeffreyip staff     721 Jan 19 06:24 deepeval-0.20.53/deepeval/metrics/base_metric.py
-rw-r--r--  0 jeffreyip staff    1430 Jan 23 02:39 deepeval-0.20.53/deepeval/metrics/bias.py
-rw-r--r--  0 jeffreyip staff    5033 Jan 19 08:47 deepeval-0.20.53/deepeval/metrics/contextual_precision.py
-rw-r--r--  0 jeffreyip staff    3735 Jan 19 08:47 deepeval-0.20.53/deepeval/metrics/contextual_recall.py
-rw-r--r--  0 jeffreyip staff    4472 Jan 19 08:47 deepeval-0.20.53/deepeval/metrics/contextual_relevancy.py
-rw-r--r--  0 jeffreyip staff     477 Jan 18 02:43 deepeval-0.20.53/deepeval/metrics/cost.py
-rw-r--r--  0 jeffreyip staff    5484 Jan 25 04:10 deepeval-0.20.53/deepeval/metrics/faithfulness.py
-rw-r--r--  0 jeffreyip staff    3987 Jan 18 04:22 deepeval-0.20.53/deepeval/metrics/g_eval.py
-rw-r--r--  0 jeffreyip staff    1116 Jan 11 23:01 deepeval-0.20.53/deepeval/metrics/hallucination_metric.py
-rw-r--r--  0 jeffreyip staff    2247 Jan 11 23:01 deepeval-0.20.53/deepeval/metrics/judgemental_gpt.py
-rw-r--r--  0 jeffreyip staff     489 Jan 18 02:43 deepeval-0.20.53/deepeval/metrics/latency.py
-rw-r--r--  0 jeffreyip staff   19085 Jan 19 06:24 deepeval-0.20.53/deepeval/metrics/ragas.py
-rw-r--r--  0 jeffreyip staff    5008 Jan 25 04:59 deepeval-0.20.53/deepeval/metrics/summarization.py
-rw-r--r--  0 jeffreyip staff    1285 Jan 23 02:39 deepeval-0.20.53/deepeval/metrics/toxicity.py
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/deepeval/models/
-rw-r--r--  0 jeffreyip staff     488 Dec 28 14:04 deepeval-0.20.53/deepeval/models/__init__.py
-rw-r--r--  0 jeffreyip staff   19602 Dec 28 14:04 deepeval-0.20.53/deepeval/models/_summac_model.py
-rw-r--r--  0 jeffreyip staff    2183 Nov 29 23:34 deepeval-0.20.53/deepeval/models/answer_relevancy_model.py
-rw-r--r--  0 jeffreyip staff     778 Nov 29 23:34 deepeval-0.20.53/deepeval/models/base.py
-rw-r--r--  0 jeffreyip staff     985 Nov 29 23:34 deepeval-0.20.53/deepeval/models/detoxify_model.py
-rw-r--r--  0 jeffreyip staff    3081 Jan 18 04:22 deepeval-0.20.53/deepeval/models/gpt_model.py
-rw-r--r--  0 jeffreyip staff     807 Nov 29 23:34 deepeval-0.20.53/deepeval/models/hallucination_model.py
-rw-r--r--  0 jeffreyip staff    2010 Jan 11 23:01 deepeval-0.20.53/deepeval/models/summac_model.py
-rw-r--r--  0 jeffreyip staff     591 Nov 29 23:34 deepeval-0.20.53/deepeval/models/unbias_model.py
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/deepeval/plugins/
-rw-r--r--  0 jeffreyip staff       0 Oct  6 07:22 deepeval-0.20.53/deepeval/plugins/__init__.py
-rw-r--r--  0 jeffreyip staff    1422 Jan 23 22:59 deepeval-0.20.53/deepeval/plugins/plugin.py
-rw-r--r--  0 jeffreyip staff    1260 Jan 18 04:22 deepeval-0.20.53/deepeval/progress_context.py
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/deepeval/scorer/
-rw-r--r--  0 jeffreyip staff      27 Nov 22 04:32 deepeval-0.20.53/deepeval/scorer/__init__.py
-rw-r--r--  0 jeffreyip staff   13743 Dec 28 14:04 deepeval-0.20.53/deepeval/scorer/scorer.py
-rw-r--r--  0 jeffreyip staff     532 Oct 22 10:37 deepeval-0.20.53/deepeval/singleton.py
-rw-r--r--  0 jeffreyip staff    1075 Jan 19 05:28 deepeval-0.20.53/deepeval/telemetry.py
-rw-r--r--  0 jeffreyip staff   17454 Jan  3 14:03 deepeval-0.20.53/deepeval/templates.py
-rw-r--r--  0 jeffreyip staff     942 Jan 18 02:43 deepeval-0.20.53/deepeval/test_case.py
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/deepeval/test_run/
-rw-r--r--  0 jeffreyip staff     125 Dec 28 14:04 deepeval-0.20.53/deepeval/test_run/__init__.py
-rw-r--r--  0 jeffreyip staff     938 Jan 19 06:38 deepeval-0.20.53/deepeval/test_run/api.py
-rw-r--r--  0 jeffreyip staff     375 Jan 23 22:59 deepeval-0.20.53/deepeval/test_run/hooks.py
-rw-r--r--  0 jeffreyip staff   11438 Jan 23 22:59 deepeval-0.20.53/deepeval/test_run/test_run.py
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/deepeval/tracing/
-rw-r--r--  0 jeffreyip staff     200 Jan 25 02:26 deepeval-0.20.53/deepeval/tracing/__init__.py
-rw-r--r--  0 jeffreyip staff    7283 Jan 16 05:28 deepeval-0.20.53/deepeval/tracing/tracing.py
-rw-r--r--  0 jeffreyip staff      95 Dec 28 14:04 deepeval-0.20.53/deepeval/types.py
-rw-r--r--  0 jeffreyip staff    5264 Jan 21 12:05 deepeval-0.20.53/deepeval/utils.py
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/deepeval.egg-info/
-rw-r--r--  0 jeffreyip staff     834 Jan 25 07:10 deepeval-0.20.53/deepeval.egg-info/PKG-INFO
-rw-r--r--  0 jeffreyip staff    2535 Jan 25 07:10 deepeval-0.20.53/deepeval.egg-info/SOURCES.txt
-rw-r--r--  0 jeffreyip staff       1 Jan 25 07:10 deepeval-0.20.53/deepeval.egg-info/dependency_links.txt
-rw-r--r--  0 jeffreyip staff      97 Jan 25 07:10 deepeval-0.20.53/deepeval.egg-info/entry_points.txt
-rw-r--r--  0 jeffreyip staff     237 Jan 25 07:10 deepeval-0.20.53/deepeval.egg-info/requires.txt
-rw-r--r--  0 jeffreyip staff      15 Jan 25 07:10 deepeval-0.20.53/deepeval.egg-info/top_level.txt
-rw-r--r--  0 jeffreyip staff    1171 Jan 25 07:09 deepeval-0.20.53/pyproject.toml
-rw-r--r--  0 jeffreyip staff      38 Jan 25 07:10 deepeval-0.20.53/setup.cfg
-rw-r--r--  0 jeffreyip staff    1166 Jan 16 05:48 deepeval-0.20.53/setup.py
drwxr-xr-x  0 jeffreyip staff       0 Jan 25 07:10 deepeval-0.20.53/tests/
-rw-r--r--  0 jeffreyip staff       0 Aug 29 11:44 deepeval-0.20.53/tests/__init__.py
-rw-r--r--  0 jeffreyip staff    3122 Jan 21 12:05 deepeval-0.20.53/tests/test_answer_relevancy.py
-rw-r--r--  0 jeffreyip staff     515 Jan 21 12:28 deepeval-0.20.53/tests/test_bias.py
-rw-r--r--  0 jeffreyip staff    3973 Jan 18 04:16 deepeval-0.20.53/tests/test_contextual_precision.py
-rw-r--r--  0 jeffreyip staff    2334 Jan 18 04:16 deepeval-0.20.53/tests/test_contextual_recall.py
-rw-r--r--  0 jeffreyip staff    2375 Jan 18 04:16 deepeval-0.20.53/tests/test_contextual_relevancy.py
-rw-r--r--  0 jeffreyip staff     288 Jan 18 02:43 deepeval-0.20.53/tests/test_cost.py
-rw-r--r--  0 jeffreyip staff    1185 Jan 18 02:43 deepeval-0.20.53/tests/test_custom_metric.py
-rw-r--r--  0 jeffreyip staff    1367 Jan 11 23:01 deepeval-0.20.53/tests/test_dataset.py
-rw-r--r--  0 jeffreyip staff    2283 Jan 19 06:32 deepeval-0.20.53/tests/test_faithfulness.py
-rw-r--r--  0 jeffreyip staff     692 Jan 25 04:59 deepeval-0.20.53/tests/test_g_eval.py
-rw-r--r--  0 jeffreyip staff    1184 Jan 23 22:59 deepeval-0.20.53/tests/test_hallucination_metric.py
-rw-r--r--  0 jeffreyip staff     767 Jan 11 23:01 deepeval-0.20.53/tests/test_judgemental.py
-rw-r--r--  0 jeffreyip staff     332 Jan 18 02:43 deepeval-0.20.53/tests/test_latency.py
-rw-r--r--  0 jeffreyip staff    1719 Jan 11 23:01 deepeval-0.20.53/tests/test_quickstart.py
-rw-r--r--  0 jeffreyip staff     777 Jan 19 08:47 deepeval-0.20.53/tests/test_rag_metrics.py
-rw-r--r--  0 jeffreyip staff    2259 Jan 19 06:24 deepeval-0.20.53/tests/test_ragas.py
-rw-r--r--  0 jeffreyip staff    9232 Dec 28 14:04 deepeval-0.20.53/tests/test_scoring.py
-rw-r--r--  0 jeffreyip staff    3836 Jan 18 03:51 deepeval-0.20.53/tests/test_summarization.py
-rw-r--r--  0 jeffreyip staff    1074 Jan 21 12:28 deepeval-0.20.53/tests/test_toxic.py
```



@nictuku I see, it seems like the latest version with the integrations directory isn't out yet. It will be shipped in the next hour or so.

@penguine-ip that's not the problem. I sent a PR to fix the real issue. Thanks for the awesome project.