Repository: explodinggradients/ragas
Issue: Ragas Synthetic Test Data Generation error using AzureOpenaiEmbeddings
Root cause: Validation Gap

I was trying to do "test generation" using RAGAS framework with the help of the "https://docs.ragas.io/en/stable/concepts/testset_generation.html", I'm facing error. 

Please have a look on the below error. 

ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
Task was destroyed but it is pending!
task: <Task pending name='Task-3' coro=<as_completed.<locals>.sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\sit
e-packages\ragas\executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]>
Task was destroyed but it is pending!
task: <Task pending name='Task-5' coro=<as_completed.<locals>.sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\sit
e-packages\ragas\executor.py:38> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]>
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<as_completed.<locals>.sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\site-packages\ragas\executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]>

And my code is: 

loader = PyPDFLoader("<some documnet> .pdf")
documents = loader.load()

#used same LLM for both generator_lmm and critic_llm
generator_llm = llm()
critic_llm = llm() 
embeddings =embeddings() # used AzureOpenaiEmbeddings

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)
distributions = {
    simple: 0.5,
    multi_context: 0.4,
    reasoning: 0.1
}


testset = generator.generate_with_langchain_docs(documents, 10, distributions)
testset.to_pandas()

versions:
openai - 1.17.0
ragas - 0.1.4/ 0.1.6

I don't think this is a Ragas issue - are you still facing this @sona-16 ?

Hi jjmachan, 

I haven't tried yet on the issue. Will do it by today and update you

I am facing this same issue.


Filename and doc_id are the same for all nodes.               
Generating:  52%|█████▎    | 21/40 [02:41<02:26,  7.69s/it]
Exception in thread Thread-21:
Traceback (most recent call last):
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/Users/L037301/Documents/GitHub/ragas/src/ragas/executor.py", line 96, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/Users/L037301/Documents/GitHub/ragas/src/ragas/executor.py", line 84, in _aresults
    raise e
  File "/Users/L037301/Documents/GitHub/ragas/src/ragas/executor.py", line 79, in _aresults
    r = await future
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 614, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/Users/L037301/Documents/GitHub/ragas/src/ragas/executor.py", line 38, in sema_coro
    return await coro
  File "/Users/L037301/Documents/GitHub/ragas/src/ragas/executor.py", line 112, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/Users/L037301/Documents/GitHub/ragas/src/ragas/testset/evolutions.py", line 144, in evolve
    return await self.generate_datarow(
  File "/Users/L037301/Documents/GitHub/ragas/src/ragas/testset/evolutions.py", line 210, in generate_datarow
    selected_nodes = [
  File "/Users/L037301/Documents/GitHub/ragas/src/ragas/testset/evolutions.py", line 213, in <listcomp>
    if i - 1 < len(current_nodes.nodes)
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'
---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
File [/Users/L037301/Documents/GitHub/ragas/src/ragas/tryout.py:1](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/src/ragas/tryout.py:1)
----> [1](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/src/ragas/tryout.py:1) testset4 = generator.generate_with_langchain_docs(pages4, test_size=40, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})

File [~/Documents/GitHub/ragas/src/ragas/testset/generator.py:179](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:179), in TestsetGenerator.generate_with_langchain_docs(self, documents, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
    [174](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:174) # chunk documents and add to docstore
    [175](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:175) self.docstore.add_documents(
    [176](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:176)     [Document.from_langchain_document(doc) for doc in documents]
    [177](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:177) )
--> [179](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:179) return self.generate(
    [180](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:180)     test_size=test_size,
    [181](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:181)     distributions=distributions,
    [182](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:182)     with_debugging_logs=with_debugging_logs,
    [183](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:183)     is_async=is_async,
    [184](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:184)     raise_exceptions=raise_exceptions,
    [185](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:185)     run_config=run_config,
    [186](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:186) )

File [~/Documents/GitHub/ragas/src/ragas/testset/generator.py:274](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:274), in TestsetGenerator.generate(self, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
    [272](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:272)     test_data_rows = exec.results()
    [273](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:273)     if not test_data_rows:
--> [274](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:274)         raise ExceptionInRunner()
    [276](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:276) except ValueError as e:
    [277](https://file+.vscode-resource.vscode-cdn.net/Users/L037301/Documents/GitHub/ragas/~/Documents/GitHub/ragas/src/ragas/testset/generator.py:277)     raise e

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

Any update? I'm facing the same message for a  ragas.evaluate inside a ThreadPoolExecutor

I also have the same issue, everything worked fine for a set of 300 pdfs, and now all of a sudden the same code gives the error below:

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

Hi Team, 

I'm still facing the same error. 
I doubt, is this due to huggingface LLM I'm using or due to computation power. 
And also, Im using google colab notebook with CPU setup to do this task. 

Hi @damlitos
could you please share me the code snippet, why because I have just used 2 pages of PDF, even though no proper output. As you told you may be getting an satisfactory answer for lesser # of pages. 

# error:

embedding nodes:   0%
 0/4 [02:27<?, ?it/s]
Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 96, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 84, in _aresults
    raise e
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 79, in _aresults
    r = await future
  File "/usr/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 38, in sema_coro
    return await coro
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 112, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ragas/embeddings/base.py", line 23, in embed_text
    embs = await self.embed_texts([text], is_async=is_async)
  File "/usr/local/lib/python3.10/dist-packages/ragas/embeddings/base.py", line 33, in embed_texts
    return await aembed_documents_with_retry(texts)
  File "/usr/local/lib/python3.10/dist-packages/tenacity/_asyncio.py", line 142, in async_wrapped
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tenacity/_asyncio.py", line 58, in __call__
    do = await self.iter(retry_state=retry_state)
  File "/usr/local/lib/python3.10/dist-packages/tenacity/_asyncio.py", line 110, in iter
    result = await action(retry_state)
  File "/usr/local/lib/python3.10/dist-packages/tenacity/_asyncio.py", line 78, in inner
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 410, in exc_check
    raise retry_exc.reraise()
  File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 183, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/dist-packages/tenacity/_asyncio.py", line 61, in __call__
    result = await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ragas/embeddings/base.py", line 64, in aembed_documents
    return await self.embeddings.aembed_documents(texts)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1709, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'SentenceTransformer' object has no attribute 'aembed_documents'
---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
[<ipython-input-26-9acca3785365>](https://localhost:8080/#) in <cell line: 1>()
----> 1 testset = generator.generate_with_langchain_docs(pages, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})

2 frames
[/usr/local/lib/python3.10/dist-packages/ragas/testset/docstore.py](https://localhost:8080/#) in add_nodes(self, nodes, show_progress)
    252         results = executor.results()
    253         if not results:
--> 254             raise ExceptionInRunner()
    255 
    256         for i, n in enumerate(nodes):

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.


Closing this as resolved in latest versions.

Ref PRs: #2070, #2286 and #2340