Repository: explodinggradients/ragas
Issue: TestsetGenerator failing to generate dataset from either LlamaIndex or LangChain documents
Root cause: Validation Gap

[X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Hi,

I'm using Ragas v0.1.3 and I'm also getting the same error `IndexError: list index out of range` regardless of generating from LangChain or LlamaIndex documents. The error seems to be related to the DocumentStore population. 

Thanks in advance for your help.

Ragas version: 0.1.3
Python version: 3.10

**Code to Reproduce**
```
from langchain_community.llms import VLLMOpenAI
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain.document_loaders import PyPDFDirectoryLoader
from langchain.text_splitter import TokenTextSplitter


from ragas.testset import TestsetGenerator
from ragas.testset.docstore import InMemoryDocumentStore
from ragas.testset.extractor import KeyphraseExtractor
from ragas.run_config import RunConfig
from ragas.embeddings.base import BaseRagasEmbeddings, LangchainEmbeddingsWrapper
from ragas.llms import BaseRagasLLM, LangchainLLMWrapper



PDF_FILES_PATH = "../../doc_collections"

# LLamaIndex embedding model
EMBEDDING_MODEL = "BAAI/bge-base-en-v1.5"
model_kwargs = {'device': 'cuda:0'}
encode_kwargs = {'normalize_embeddings': False}
langchain_emb = HuggingFaceEmbeddings(
    model_name=EMBEDDING_MODEL,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs
)

# LLM service settings
langchain_llm = VLLMOpenAI(
    openai_api_base="localhost/api/v1",
    model_name="WizardLM/WizardLM-70B-V1.0",
    temperature=0.1,
    max_tokens=512,
)

langchain_llm = LangchainLLMWrapper(langchain_llm)
langchain_emb = LangchainEmbeddingsWrapper(langchain_emb)


loader = PyPDFDirectoryLoader(path=PDF_FILES_PATH)
lchain_docs = loader.load()


# Add custom llms and embeddings
generator_llm = langchain_llm
critic_llm = langchain_llm

splitter = TokenTextSplitter(chunk_size=1024, chunk_overlap=20)
keyphrase_extractor = KeyphraseExtractor(llm=critic_llm)
doc_store = InMemoryDocumentStore(
    splitter=splitter,
    embeddings=langchain_emb,
    extractor=keyphrase_extractor,
)
dataset_distribution = {
    "simple": 0.5,
    "reasoning": 0.25,
    # "multi_context": 0.1,
    "conditional": 0.25,
}

dataset_generator = TestsetGenerator(
    generator_llm=generator_llm,
    critic_llm=critic_llm,
    embeddings=langchain_emb,
    docstore=doc_store,
)


# RunConfig.max_workers=4
dataset_generator.generate_with_langchain_docs(
    documents=lchain_docs[:50],
    test_size=10,
    distributions=dataset_distribution,
 #   run_config=RunConfig
)   

```



**Error trace**
```
ExceptionInRunner                         Traceback (most recent call last)
Cell In[3], line 32
     23 dataset_generator = TestsetGenerator(
     24     generator_llm=generator_llm,
     25     critic_llm=critic_llm,
     26     embeddings=langchain_emb,
     27     docstore=doc_store,
     28 )
     31 # RunConfig.max_workers=4
---> 32 dataset_generator.generate_with_langchain_docs(
     33     documents=lchain_docs[:50],
     34     test_size=10,
     35     distributions=dataset_distribution,
     36  #   run_config=RunConfig
     37 )   

File ~/miniconda3/envs/llm-env4/lib/python3.10/site-packages/ragas/testset/generator.py:152, in TestsetGenerator.generate_with_langchain_docs(self, documents, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
    141 def generate_with_langchain_docs(
    142     self,
    143     documents: t.Sequence[LCDocument],
   (...)
    150 ):
    151     # chunk documents and add to docstore
--> 152     self.docstore.add_documents(
    153         [Document.from_langchain_document(doc) for doc in documents]
    154     )
    156     return self.generate(
    157         test_size=test_size,
    158         distributions=distributions,
   (...)
    162         run_config=run_config,
    163     )

File ~/miniconda3/envs/llm-env4/lib/python3.10/site-packages/ragas/testset/docstore.py:215, in InMemoryDocumentStore.add_documents(self, docs, show_progress)
    210 # split documents with self.splitter into smaller nodes
    211 nodes = [
    212     Node.from_langchain_document(d)
    213     for d in self.splitter.transform_documents(docs)
    214 ]
--> 215 self.add_nodes(nodes, show_progress=show_progress)

File ~/miniconda3/envs/llm-env4/lib/python3.10/site-packages/ragas/testset/docstore.py:254, in InMemoryDocumentStore.add_nodes(self, nodes, show_progress)
    252 results = executor.results()
    253 if results == []:
--> 254     raise ExceptionInRunner()
    256 for i, n in enumerate(nodes):
    257     if i in nodes_to_embed.keys():

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
```



**Expected behavior**
To get the test dataset generated.

**Additional context**
Seems to be the same issue reported very recently by multiple people at #563 even after being declared as fixed. Thanks!


<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Just tried the same code and this error went away with Ragas 0.1.4 but other errors showed up. I'll open a new issue for that. Thanks!