Repository: explodinggradients/ragas
Issue: Event loop stopped before Future completed -generator.generate_with_langchain_docs
Root cause: Resource Mishandling

ragas                                    0.2.3
langchain                                0.2.16
langchain-chroma                         0.1.4
langchain-community                      0.2.16
langchain-core                           0.2.40
langchain-huggingface                    0.1.0
langchain-openai                         0.1.23
langchain-text-splitters                 0.2.4
langchain-unstructured                   0.1.4
![image](https://github.com/user-attachments/assets/9013793d-8e87-4142-ac87-d8af76105a8f)


`File "/home/alg/qdl/evalscope/package/evalscope/backend/rag_eval/backend_manager.py", line 54, in run_ragas
    generate_testset(TestsetGenerationArguments(**testset_args))
  File "/home/alg/qdl/evalscope/package/evalscope/backend/rag_eval/ragas/tasks/testset_generation.py", line 167, in generate_testset
    testset = generator.generate_with_langchain_docs(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/generate.py", line 129, in generate_with_langchain_docs
    return self.generate(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/generate.py", line 254, in generate
    raise e
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/generate.py", line 251, in generate
    scenario_sample_list: t.List[t.List[BaseScenario]] = exec.results()
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 146, in results
    results = asyncio.run(_aresults())
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 141, in _aresults
    r = await future
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 36, in sema_coro
    return await coro
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 81, in wrapped_callable_async
    raise e
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 78, in wrapped_callable_async
    result = await callable(*args, **kwargs)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/base.py", line 89, in generate_scenarios
    scenarios = await self._generate_scenarios(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/abstract_query.py", line 263, in _generate_scenarios
    common_concepts: t.List[Concepts] = run_async_batch(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 167, in run_async_batch
    return executor.results()
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 146, in results
    results = asyncio.run(_aresults())
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/nest_asyncio.py", line 30, in run
    return loop.run_until_complete(task)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/nest_asyncio.py", line 98, in run_until_complete
    return f.result()
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 141, in _aresults
    r = await future
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 36, in sema_coro
    return await coro
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 81, in wrapped_callable_async
    raise e
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 78, in wrapped_callable_async
    result = await callable(*args, **kwargs)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/prompt/pydantic_prompt.py", line 130, in generate
    output_single = await self.generate_multiple(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/prompt/pydantic_prompt.py", line 190, in generate_multiple
    resp = await llm.generate(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/llms/base.py", line 100, in generate
    result = await agenerate_text_with_retry(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/llms/base.py", line 220, in agenerate_text
    return await self.langchain_llm.agenerate_prompt(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 787, in agenerate_prompt
    return await self.agenerate(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 747, in agenerate
    raise exceptions[0]
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 923, in _agenerate_with_cache
    result = await self._agenerate(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/langchain_openai/chat_models/base.py", line 815, in _agenerate
    response = await self.async_client.create(**payload)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/openai/resources/chat/completions.py", line 1490, in create
    return await self._post(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/openai/_base_client.py", line 1841, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/openai/_base_client.py", line 1533, in request
    return await self._request(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/openai/_base_client.py", line 1608, in _request
    raise APIConnectionError(request=request) from err
openai.APIConnectionError: Connection error.
2024-10-31 17:28:08,770 - asyncio - ERROR - Task exception was never retrieved
future: <Task finished name='Task-334' coro=<as_completed.<locals>.sema_coro() done, defined at /home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py:34> exception=RuntimeError('Event loop stopped before Future completed.')>
Traceback (most recent call last):
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 36, in sema_coro
    return await coro
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 81, in wrapped_callable_async
    raise e
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 78, in wrapped_callable_async
    result = await callable(*args, **kwargs)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/base.py", line 89, in generate_scenarios
    scenarios = await self._generate_scenarios(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/abstract_query.py", line 119, in _generate_scenarios
    themes: t.List[Themes] = run_async_batch(
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 167, in run_async_batch
    return executor.results()
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 146, in results
    results = asyncio.run(_aresults())
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/nest_asyncio.py", line 30, in run
    return loop.run_until_complete(task)
  File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/nest_asyncio.py", line 96, in run_until_complete
    raise RuntimeError(
RuntimeError: Event loop stopped before Future completed.
`


The generator llm is  remote api model,and the embeddings is local model

@jackqdldd is it a huggingface model?

> @jackqdldd is it a huggingface model?

no,the llm is Qwen2.5-72B-Instruct-AWQ from modelscope

![image](https://github.com/user-attachments/assets/1dff3803-2cb0-4ace-a9c7-75a1a0ef641e)

it seems the [Generating Scenarios:] was interrupted