Repository: confident-ai/deepeval
Issue: AssertionError: n_contexts_per_doc must be a positive integer.
Root cause: Validation Gap

**Discord**
This question already ask in discord. Thank you.

**Describe the bug**
I try to use **Synthesizer** to create dataset, but facing problem like this.
```
/tmp2/chten/miniforge3/envs/delta/lib/python3.12/site-packages/deepeval/__init__.py:53: UserWarning: You are using deepeval version 1.5.9, however version 2.0 is available. You should consider upgrading via the "pip install --upgrade deepeval" command.
  warnings.warn(
✨ 🚀 ✨ Loading Documents: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 11.70it/s]
✨ 🧩 ✨ Generating Contexts: 0it [00:00, ?it/s]Traceback (most recent call last):
  File "/tmp2/chten/delta/DeepEval/synthesizer/test.py", line 19, in <module>
    synthesizer.generate_goldens_from_docs(
  File "/tmp2/chten/miniforge3/envs/delta/lib/python3.12/site-packages/deepeval/synthesizer/synthesizer.py", line 122, in generate_goldens_from_docs
    goldens = loop.run_until_complete(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp2/chten/miniforge3/envs/delta/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/tmp2/chten/miniforge3/envs/delta/lib/python3.12/site-packages/deepeval/synthesizer/synthesizer.py", line 202, in a_generate_goldens_from_docs
    await self.context_generator.a_generate_contexts(
  File "/tmp2/chten/miniforge3/envs/delta/lib/python3.12/site-packages/deepeval/synthesizer/chunking/context_generator.py", line 194, in a_generate_contexts
    results = await asyncio.gather(*tasks)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp2/chten/miniforge3/envs/delta/lib/python3.12/site-packages/deepeval/synthesizer/chunking/context_generator.py", line 217, in _a_process_document_async
    await self._a_get_n_random_contexts_per_doc(
  File "/tmp2/chten/miniforge3/envs/delta/lib/python3.12/site-packages/deepeval/synthesizer/chunking/context_generator.py", line 295, in _a_get_n_random_contexts_per_doc
    n_contexts_per_doc > 0
AssertionError: n_contexts_per_doc must be a positive integer.
✨ 🧩 ✨ Generating Contexts: 0it [00:01, ?it/s]
```

**To Reproduce**
This is the code I run, and the **leed20.docx** is a doc about 51 pages.
I already try for setting **chunk size** to 500, 200, 64 and 15, but all these chunk size I try also get the same error.
The chroma version is v0.5.3. 

```python
from deepeval.synthesizer import Synthesizer
from deepeval.dataset import EvaluationDataset
from deepeval.synthesizer.config import ContextConstructionConfig

dataset = EvaluationDataset()
synthesizer = Synthesizer()
context_config=ContextConstructionConfig(
    # embedder=None,
    # critic_model=None,
    max_contexts_per_document=1,
    chunk_size=1000,
    chunk_overlap=0,
    context_quality_threshold=0.5,
    context_similarity_threshold=0.5,
    max_retries=1)

synthesizer.generate_goldens_from_docs(
    document_paths=['leed20.docx'],
    include_expected_output=True,
    max_goldens_per_context=1,
    context_construction_config=context_config
)
print(synthesizer.synthetic_goldens)
dataframe = synthesizer.to_pandas()
print(dataframe)
```

**Desktop:**
 - Ubuntu 22.04.5 LTS (GNU/Linux 5.15.0-126-generic x86_64)

**LLM setting**
- deepeval set-local-model --model-name="llama3.2:3b" --base-url="http://localhost:11434/" --api-key="ollama"
- deepeval set-local-embeddings --model-name="nomic-embed-text:latest" --base-url="http://localhost:11434/" --api-key="ollama"



I have the same error ... `n_contexts_per_doc must be a positive integer`. When I slightly increase or decrease the chunk_size, it might suddenly work. However, after it worked and I run it again with the same chunk_size, the error `n_contexts_per_doc must be a positive integer` appears again. Are there any workarounds how to prevent this error?

deepeval version: 2.0.5


The synthesizer will raise an error if chunk_size is too large to generate n=max_contexts_per_document unique contexts.
@tenchanhe 


i ran into the same error and my chunk size is only 256 

Is there any update to this issue?

An update on this issue. I dont see this issue now even with the default chunk size of 1024. I use chroma and amazonbedrock for custom llm.

i have the same issue when i am tying to use custom llm deepseek and any embedding model

The issue is still actual.
When trying to set a smaller chunk size (e.g. 512), then getting another error:

                                                                    python .\scripts\deep_eval_data_generator.py
✨ 🚀 ✨ Loading Documents: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.31s/it]
✨ 📚 ✨ Chunking Documents:   0%|                                                                                                                                                                           | 0/1 [00:00<?, ?it/s]T Traceback (most recent call last):
  File "C:\Users\x\repos\insights-data-generator\scripts\deep_eval_data_generator.py", line 36, in <module>
    goldens = synthesizer.generate_goldens_from_docs(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\x\repos\insights-data-generator\.venv\Lib\site-packages\deepeval\synthesizer\synthesizer.py", line 124, in generate_goldens_from_docs
    goldens = loop.run_until_complete(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\asyncio\base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\Users\x\repos\insights-data-generator\.venv\Lib\site-packages\deepeval\synthesizer\synthesizer.py", line 208, in a_generate_goldens_from_docs
    await self.context_generator.a_generate_contexts(
  File "C:\Users\x\repos\insights-data-generator\.venv\Lib\site-packages\deepeval\synthesizer\chunking\context_generator.py", line 175, in a_generate_contexts
    await tqdm_asyncio.gather(
  File "C:\Users\x\repos\insights-data-generator\.venv\Lib\site-packages\tqdm\asyncio.py", line 79, in gather
    res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\x\repos\insights-data-generator\.venv\Lib\site-packages\tqdm\asyncio.py", line 79, in <listcomp>
    res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout,
           ^^^^^^^
  File "C:\Program Files\Python311\Lib\asyncio\tasks.py", line 615, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "C:\Users\x\repos\insights-data-generator\.venv\Lib\site-packages\tqdm\asyncio.py", line 76, in wrap_awaitable
    return i, await f
              ^^^^^^^
  File "C:\Users\x\repos\insights-data-generator\.venv\Lib\site-packages\deepeval\synthesizer\chunking\context_generator.py", line 165, in a_chunk_and_store
    await chunker.a_chunk_doc()
  File "C:\Users\x\repos\insights-data-generator\.venv\Lib\site-packages\deepeval\synthesizer\chunking\doc_chunker.py", line 62, in a_chunk_doc
    collection = client.get_collection(name=collection_name)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\x\repos\insights-data-generator\.venv\Lib\site-packages\chromadb\api\client.py", line 132, in get_collection
    return self._server.get_collection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\x\repos\insights-data-generator\.venv\Lib\site-packages\chromadb\telemetry\opentelemetry\__init__.py", line 146, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\x\repos\insights-data-generator\.venv\Lib\site-packages\chromadb\api\segment.py", line 262, in get_collection
    raise ValueError(f"Collection {name} does not exist.")
ValueError: Collection processed_chunks_512_0 does not exist.

@tenchanhe Just to confirm, could you try deleting the .vector_db folder and rerunning the synthesizer? I suspect the issue is that the embedding model isn't properly supporting Ollama. As a result, the first time you run the code, the error you encounter isn't actually related to n_context_per_doc. Instead, it's likely a 404 error when attempting to connect to your local model. But the error afterwards will display the n_context_per_doc message.

> [@tenchanhe](https://github.com/tenchanhe) Just to confirm, could you try deleting the .vector_db folder and rerunning the synthesizer? I suspect the issue is that the embedding model isn't properly supporting Ollama. As a result, the first time you run the code, the error you encounter isn't actually related to n_context_per_doc. Instead, it's likely a 404 error when attempting to connect to your local model. But the error afterwards will display the n_context_per_doc message.

Hi, I have already received this instruction on Discord to delete the .vector_db folder and rerun, but the issue still persists.

<img width="1421" alt="Image" src="https://github.com/user-attachments/assets/a3cd5c20-5c26-4db1-8c5e-58254a400833" />

@kritinv 
Is anyone able to create Datasets after getting this issue

As Iam getting the below error where it is not able to chunk the data
Always getting num_chunks to be 0 after synthesizing a pdf document


Please ping if there is any update or workaround for  the above issue

This issuse is caused by wrong embed config,  The following is correct: 

```
"USE_LOCAL_EMBEDDINGS": "YES",
    "LOCAL_EMBEDDING_MODEL_NAME": "nomic-embed-text:latest",
    "LOCAL_EMBEDDING_API_KEY": "ollama",
    "LOCAL_EMBEDDING_BASE_URL": "http://localhost:11434/v1/"
```

It's  "http://localhost:11434/v1/",  NOT "http://localhost:11434"

My team is also getting this. We are using Gemini through VertexAI with text-embedding-004. We already chunk these using a token text splitter and haven't had any issues, but during the deep eval we are getting this. We are getting it even at 250 chunk size. 

Further more, I'm confused by the error message warning in the docs https://docs.confident-ai.com/docs/synthesizer-generate-from-docs#customize-context-construction . 

If we had a file that is < 250 would it just take all from the 1-250? Or would it error?