Repository: confident-ai/deepeval
Issue: Synthesizer ChromaDB - no required index generated
Root cause: Validation Gap

**Describe the bug**
Following the guide to synthesize my goldens [here](https://docs.confident-ai.com/docs/synthesizer-generate-from-docs). Installing DeepEval and ChromaDB by following [this](https://docs.confident-ai.com/docs/getting-started) guide and doing a `pip install chromadb==0.5.3` respectively, into the virtualenv. Reading a dataset will throw an error about ChromaDB not having had created the necessary indexes. There's nothing in the user guide about creating indexes before running the synthesizer by hand. I'd expect DeepEval to create the necessary ChromaDB indexes in the background automatically, OR to have the requirements about how to do this (what's the expectation for the index name for example?) detailed in the docs.

This is on:
pip show deepeval
Name: deepeval
Version: 2.5.3
Summary: The Open-Source LLM Evaluation Framework.
Home-page: https://github.com/confident-ai/deepeval

**To Reproduce**
Steps to reproduce the behavior:
1. Set up DeepEval in a venv as described [here](https://docs.confident-ai.com/docs/getting-started).
2. Install ChromaDB as described in the [docs](https://docs.confident-ai.com/docs/synthesizer-generate-from-docs) via `pip install chromadb==0.5.3`
3. Try to run the following example code:

```
from deepeval.synthesizer import Synthesizer
from pathlib import Path
from dotenv import load_dotenv
from deepeval.synthesizer.config import ContextConstructionConfig
load_dotenv()

document_paths = [str(p) for p in Path("path_to_test_pdfs").rglob("*.pdf")]
synthesizer = Synthesizer()
print("Starting to generate goldens")
synthesizer.generate_goldens_from_docs(
    document_paths=document_paths,
    max_goldens_per_context=2,
    context_construction_config=ContextConstructionConfig(
        chunk_size=363
    )
)

print("Finished generating goldens")
synthesizer.save_as(
    file_type='csv',
    directory="./synthetic_data"
)
```

4. See error:
```
Starting to generate goldens
✨ 🚀 ✨ Loading Documents: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:04<00:00,  3.30it/s]
✨ 📚 ✨ Chunking Documents:   0%|                                                                                                                                                | 0/14 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "path/generate_goldens.py", line 10, in <module>
    synthesizer.generate_goldens_from_docs(
  File "path/.venv/lib/python3.12/site-packages/deepeval/synthesizer/synthesizer.py", line 127, in generate_goldens_from_docs
    goldens = loop.run_until_complete(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/csanadpoda/.pyenv/versions/3.12.8/lib/python3.12/asyncio/base_events.py", line 686, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "path/.venv/lib/python3.12/site-packages/deepeval/synthesizer/synthesizer.py", line 219, in a_generate_goldens_from_docs
    await self.context_generator.a_generate_contexts(
  File "path/.venv/lib/python3.12/site-packages/deepeval/synthesizer/chunking/context_generator.py", line 188, in a_generate_contexts
    await tqdm_asyncio.gather(
  File "path/.venv/lib/python3.12/site-packages/tqdm/asyncio.py", line 79, in gather
    res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout,
           ^^^^^^^
  File "/Users/csanadpoda/.pyenv/versions/3.12.8/lib/python3.12/asyncio/tasks.py", line 631, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "path/.venv/lib/python3.12/site-packages/tqdm/asyncio.py", line 76, in wrap_awaitable
    return i, await f
              ^^^^^^^
  File "path/.venv/lib/python3.12/site-packages/deepeval/synthesizer/chunking/context_generator.py", line 178, in a_chunk_and_store
    await chunker.a_chunk_doc(self.chunk_size, self.chunk_overlap)
  File "path/.venv/lib/python3.12/site-packages/deepeval/synthesizer/chunking/doc_chunker.py", line 64, in a_chunk_doc
    collection = client.get_collection(name=collection_name)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path/.venv/lib/python3.12/site-packages/chromadb/api/client.py", line 132, in get_collection
    return self._server.get_collection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path/.venv/lib/python3.12/site-packages/chromadb/telemetry/opentelemetry/__init__.py", line 146, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "path/.venv/lib/python3.12/site-packages/chromadb/api/segment.py", line 262, in get_collection
    raise ValueError(f"Collection {name} does not exist.")
ValueError: Collection processed_chunks_363_0 does not exist.
Task exception was never retrieved
future: <Task finished name='Task-17' coro=<tqdm_asyncio.gather.<locals>.wrap_awaitable() done, defined at path/.venv/lib/python3.12/site-packages/tqdm/asyncio.py:75> exception=ValueError('Collection processed_chunks_363_0 does not exist.')>

```
For each file.

Funnily enough, the .vector_db folder DOES get populated, it created a folder for each file named the original file name, and there's a .vector_db/sourcefilename/chroma.sqlite3 file in each of them.

**Expected behavior**
The ChromaDB database creates the appropriate indexes by default and uses those, synthetic goldens get created successfully.

**Screenshots**
Not applicable

**Desktop (please complete the following information):**
 - OS: MacOS
 - Browser N/A, doing this in VS Code
 - Version: MacOS Sequoia 15.3.1 (24D70)

The issue seems to be that in `a_chunk_doc` you're expecting the client to throw an `InvalidCollectionException`, but in fact it throws a `ValueError`, so the index never gets created.

I'm facing the same issue, did you manage to solve it?

I encountered the same issue. After updating the Chroma version to the latest (0.6.3), it started working. The documentation might be incorrect.

We're working on removing the indexing, because there have been multiple similar issues in the past, and they have not been resolved.

> We're working on removing the indexing, because there have been multiple similar issues in the past, and they have not been resolved.

Why would you remove it? All you need to do is to catch the correct exception, then it works. (Or, not rely on tripping an exception for creating the indexes in the first place...)

Can confirm upgrading chromadb to 0.6.3 resolves this issue. Would be nice to update the documentation so it does not suggest installing `pip install chromadb==0.5.3`




> I'm facing the same issue, did you manage to solve it?

Yes, as I said:

> The issue seems to be that in `a_chunk_doc` you're expecting the client to throw an `InvalidCollectionException`, but in fact it throws a `ValueError`, so the index never gets created.



The chroma documentation has been updated. The new synthesizer w/o caching is out. @csanadpoda, yes correcting the exception will work for you, but other people have run into other issues with the synthesizer due to caching that doesn't concern catching this specific error. Should there be enough demand, we will add an improved version of caching back to the synthesizer. It is also possible to generate goldens directly from context, which gives you much more flexibility in chunking and context generation.