Repository: explodinggradients/ragas
Issue: generate_with_llamaindex_docs does not run to completion on v0.1.1
Root cause: Algorithmic Error

**Describe the bug**

`generate_with_llamaindex_docs` appears to have a regression in v0.1.1 and does not run to completion. The issue does not appear on v0.1.0

Ragas version: 0.1.1
Python version: 3.10

**Code to Reproduce**

Run the notebook in this [PR](https://github.com/explodinggradients/ragas/pull/594).

**Error trace**

```
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/executor.py", line 75, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/executor.py", line 63, in _aresults
    raise e
  File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/executor.py", line 58, in _aresults
    r = await future
  File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/executor.py", line 91, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 161, in evolve
    ) = await self._aevolve(current_tries, current_nodes)
  File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 507, in _aevolve
    result = await self._acomplex_evolution(
  File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 380, in _acomplex_evolution
    current_nodes = self.se._get_more_adjacent_nodes(current_nodes)
  File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 128, in _get_more_adjacent_nodes
    prev_adjacent_node = self.docstore.get_adjacent(
  File "/Users/xandersong/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/docstore.py", line 300, in get_adjacent
    index = self.nodes.index(node)
ValueError: Node(page_content='complex reasoning by decomposing input-output\nmappings into many intermediate steps. There are\nmany pieces of research on CoT prompting strate-\ngies (Qiao et al., 2022) including prompt designing\nand process optimization. In this paper, we mainly\nfocus on CoT designing strategies.\nSimilar to demonstration selection, CoT design-\ning also considers CoT selection. Different from\nWei et al. (2022c) manually writing CoTs, Auto-\nCoT (Zhang et al., 2022b) used LLMs with Let's\nthink step by step to generate CoTs. In addi-\ntion, Fu et al. (2022) proposed a complexity-based\ndemonstration selection method. They selected\ndemonstrations with more reasoning steps for CoT\nprompting.\nAs input-output mappings are decomposed into\nstep-by-step reasoning, some researchers apply\nmulti-stage ICL for CoT prompting and design\nCoT demonstrations for each step. Multi-stage\nICL queries LLMs with different demonstrations in\neach reasoning step. Self-Ask (Press et al., 2022) al-\nlows LLMs to generate follow-up questions for the\ninput and ask themselves these questions. Then the\nquestions and intermediate answers will be added\nto CoTs. iCAP (Wang et al., 2022a) proposes a\ncontext-aware prompter that can dynamically ad-\njust contexts for each reasoning step. Least-to-\nMost Prompting (Zhou et al., 2022a) is a two-stage\nICL including question reduction and subquestion\nsolution. The first stage decomposes a complex\nquestion into subquestions; in the second stage,\nLLMs answer subquestions sequentially, and previ-\nously answered questions and generated answers\nwill be added into the context.\nXu et al. (2023b) fine-tuned small LMs on spe-\ncific task as plug-ins to generate pseudo reasoning\nsteps. Given an input-output pair (xi, yi), Super-\nICL regarded the prediction y′\niand confidence ci\nof small LMs for the input xias reasoning steps by\nconcatenating (xi, y′\ni, ci, yi).\n3Takeaway :(1) Demonstration selection\nstrategies improve the I', metadata={'page_label': '6', 'file_name': '2301.00234v3.A_Survey_on_In_context_Learning.pdf', 'file_path': 'prompt-engineering-papers/2301.00234v3.A_Survey_on_In_context_Learning.pdf', 'file_type': 'application/pdf', 'file_size': 4898135, 'creation_date': '2024-02-13', 'last_modified_date': '2024-02-08', 'last_accessed_date': '2024-02-13'}, doc_id='963d6900-8fd2-4b7e-bfd2-00d8f29f635e', filename='963d6900-8fd2-4b7e-bfd2-00d8f29f635e') is not in list 

---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
Cell In[12], line 13
     10 distribution = {simple: 0.5, reasoning: 0.25, multi_context: 0.25}
     12 # generate testset
---> 13 testset = generator.generate_with_llamaindex_docs(
     14     documents, test_size=TEST_SIZE, distributions=distribution
     15 )
     16 test_df = testset.to_pandas()
     17 test_df.head()

File ~/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/generator.py:128, in TestsetGenerator.generate_with_llamaindex_docs(self, documents, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
    113 def generate_with_llamaindex_docs(
    114     self,
    115     documents: t.Sequence[LlamaindexDocument],
   (...)
    122 ):
    123     # chunk documents and add to docstore
    124     self.docstore.add_documents(
    125         [Document.from_llamaindex_document(doc) for doc in documents]
    126     )
--> 128     return self.generate(
    129         test_size=test_size,
    130         distributions=distributions,
    131         with_debugging_logs=with_debugging_logs,
    132         is_async=is_async,
    133         run_config=run_config,
    134         raise_exceptions=raise_exceptions,
    135     )

File ~/miniforge3/envs/ragas-test/lib/python3.10/site-packages/ragas/testset/generator.py:246, in TestsetGenerator.generate(self, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
    244     test_data_rows = exec.results()
    245     if test_data_rows == []:
--> 246         raise ExceptionInRunner()
    248 except ValueError as e:
    249     raise e

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exception=False` incase you want to show only a warning message instead.
```

**Expected behavior**
Method should run without error.


llamaindex seems broken in many ways with the latest update

Hey @ogencoglu @axiomofjoy I have raised a fix for this issue #617