Repository: explodinggradients/ragas
Issue: Task was destroyed but it is pending!
Root cause: Resource Mishandling

**Describe the bug**
I use ragas to evaluate my project,the dataset comes from the project, when I use 100 number of data,it is normal and could get the evaluate result.Then when I use more than 300 number of data,it raise exception as follow. I suspect there is something wrong with asyncio in ragas framework.

Exception ignored in: <coroutine object Executor.wrap_callable_with_index.<locals>.wrapped_callable_async at 0x7fc4761f9310>
Traceback (most recent call last):
  File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/executor.py", line 91, in wrapped_callable_async
  File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/metrics/base.py", line 91, in ascore
  File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/metrics/base.py", line 87, in ascore
  File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/metrics/_context_precision.py", line 129, in _ascore
  File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/llms/base.py", line 92, in generate
  File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
  File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/tenacity/_asyncio.py", line 47, in __call__
  File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/tenacity/__init__.py", line 334, in iter
KeyError: 'idle_for'
........
Task was destroyed but it is pending!
task: <Task pending name='context_recall-316' coro=<Executor.wrap_callable_with_index.<locals>.wrapped_callable_async() done, defined at /home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/executor.py:90> wait_for=<Future cancelled> cb=[as_completed.<locals>._on_completion() at /home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/asyncio/tasks.py:558]>
Task was destroyed but it is pending!
task: <Task pending name='context_precision-316' coro=<Executor.wrap_callable_with_index.<locals>.wrapped_callable_async() done, defined at /home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/executor.py:90> wait_for=<Future cancelled> cb=[as_completed.<locals>._on_completion() at /home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/asyncio/tasks.py:558]>


Ragas version:  0.1.2
Python version: 3.10.9

**Code to Reproduce**
import nest_asyncio
nest_asyncio.apply()

def run_evaluate(dataset,llm):
    result = evaluate(
        dataset=dataset,
        metrics=metrics,
        llm=llm,
        embeddings=build_embeddings(model_name),
        is_async=True,
        raise_exceptions=True,
    )

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


same error here

Same here!

Same here as well!

Same here :/

same

still same error....

I was trying to do "test generation" using RAGAS framework with the help of the "https://docs.ragas.io/en/stable/concepts/testset_generation.html", I'm facing error. Please have a look on the below error. 
ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
Task was destroyed but it is pending!
task: <Task pending name='Task-3' coro=<as_completed.<locals>.sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\sit
e-packages\ragas\executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]>
Task was destroyed but it is pending!
task: <Task pending name='Task-5' coro=<as_completed.<locals>.sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\sit
e-packages\ragas\executor.py:38> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]>
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<as_completed.<locals>.sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\site-packages\ragas\executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]>


Same here running  enerator.generate_with_langchain_docs

```python
from langchain_community.llms import VLLM

generator_llm = VLLM(model=args.generator,
                    trust_remote_code=True,  # mandatory for hf models
                    max_new_tokens=128,
                    top_k=10,
                    top_p=0.95,
                    temperature=0.8,
                    dtype="half",
                    enforce_eager=True
                            )

critic_llm = generator_llm.copy()

embeddings = HuggingFaceEmbeddings(model_name=args.embeddings)
generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)

    
eval_ds = generator.generate_with_langchain_docs(
    docs,
    test_size=10
    distributions={
        simple: 0.4,
        reasoning: 0.4,
        multi_context: 0.3,
    },
)
eval_ds.to_pandas().to_csv(args.output)
```



```text
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py", line 96, in run
    results = self.loop.run_until_complete(self._aresults())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py", line 84, in _aresults
    raise e
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py", line 79, in _aresults
    r = await future
        ^^^^^^^^^^^^
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py", line 605, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py", line 38, in sema_coro
    return await coro
           ^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/testset/extractor.py", line 49, in extract
    results = await self.llm.generate(prompt=prompt, is_async=is_async)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/llms/base.py", line 92, in generate
    return await agenerate_text_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 47, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 325, in iter
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 158, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 50, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/llms/base.py", line 177, in agenerate_text
    result = await self.langchain_llm.agenerate_prompt(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 643, in agenerate_prompt
    return await self.agenerate(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 1018, in agenerate
    output = await self._agenerate_helper(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 882, in _agenerate_helper
    raise e
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 866, in _agenerate_helper
    await self._agenerate(
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 544, in _agenerate
    return await run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/runnables/config.py", line 514, in run_in_executor
    return await asyncio.get_running_loop().run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_community/llms/vllm.py", line 132, in _generate
    outputs = self.client.generate(prompts, sampling_params)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 190, in generate
    return self._run_engine(use_tqdm)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 218, in _run_engine
    step_outputs = self.llm_engine.step()
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 683, in step
    return self._process_model_outputs(output, scheduler_outputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 601, in _process_model_outputs
    self._process_sequence_group_outputs(seq_group, outputs)
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 436, in _process_sequence_group_outputs
    parent_child_dict[sample.parent_seq_id].append(sample)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyError: 139
Traceback (most recent call last):
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/./src/evaluation/RAGAS.py", line 205, in <module>
    main(args, True)
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/./src/evaluation/RAGAS.py", line 111, in main
    eval_ds = generator.generate_with_langchain_docs(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/testset/generator.py", line 175, in generate_with_langchain_docs
    self.docstore.add_documents(
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/testset/docstore.py", line 215, in add_documents
    self.add_nodes(nodes, show_progress=show_progress)
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/testset/docstore.py", line 254, in add_nodes
    raise ExceptionInRunner()
ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.


Processed prompts: 2it [00:07,  3.56s/it]                       [A
Processed prompts: 3it [00:07,  2.41s/it]

Processed prompts:   0%|          | 0/3 [00:05<?, ?it/s]
Exception ignored in: <coroutine object Executor.wrap_callable_with_index.<locals>.wrapped_callable_async at 0x2acc911eee40>
Traceback (most recent call last):
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/testset/extractor.py", line 49, in extract
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/llms/base.py", line 92, in generate
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 47, in __call__
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 325, in iter
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 158, in reraise
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/concurrent/futures/_base.py", line 449, in result
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 50, in __call__
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/llms/base.py", line 177, in agenerate_text
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 643, in agenerate_prompt
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 1018, in agenerate
RuntimeError: coroutine ignored GeneratorExit
Task was destroyed but it is pending!
task: <Task pending name='Task-8' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-22' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-10' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-4' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-24' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-12' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-26' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-14' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-6' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/futures.py:387, Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>


```

I experienced the same issue. After some time of debugging, I found that leaving out this import line of another module that I used elsewhere in the code solved the issue: `from test_app import run`. No idea how and why this import line caused this error you all describe above. 

Experienced the same problem as well.

Same error here!

Same error here!

same error

Closing this as solved in recent versions.