Repository: explodinggradients/ragas
Issue: Testset generation error on Bedrock
Root cause: External Dependency Breakage

**Describe the bug**
A clear and concise description of what the bug is.

Ragas version: 0.1.0rc2.dev26+g6a88465
Python version: 3.10.13

**Code to Reproduce**
```py
###Add custom llms and embeddings
generator_llm = LangchainLLMWrapper(candidate_llm1)
critic_llm = LangchainLLMWrapper(judge_llm)
embeddings_model = LangchainEmbeddingsWrapper(titan_embeddings,RunConfig(max_retries=2, max_wait=30))

Change resulting question type distribution
testset_distribution = {
"simple": 0.25,
"reasoning": 0.5,
"multi_context": 0.0,
"conditional": 0.25,
}

splitter = TokenTextSplitter(chunk_size=1000, chunk_overlap=100)
keyphrase_extractor = keyphraseExtractor(llm=generator_llm)

docstore = InMemoryDocumentStore(
splitter=splitter,
embeddings=embeddings_model,
extractor=keyphrase_extractor,
)
test_generator = TestsetGenerator(
generator_llm=generator_llm,
critic_llm=critic_llm,
embeddings=titan_embeddings,
docstore=docstore
)

testset = test_generator.generate_with_langchain_docs(documents=documents,
test_size=10,
distributions=testset_distribution)
```

**Error trace**
```
/ragas/src/ragas/testset/docstore.py:250, in InMemoryDocumentStore.add_nodes(self, nodes, show_progress, desc)
248 for i, n in enumerate(nodes):
249 if i in nodes_to_embed.keys():
--> 250 n.embedding = results[nodes_to_embed[i]]
251 if i in nodes_to_extract.keys():
252 keyphrases = results[nodes_to_extract[i]]

IndexError: list index out of range
```

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
langchain documents.
Tried with diffrent chunk sizes and overlapps, different embedding configs and different bedrock models(cohere and titan text)
I am getting with any model on Bedrock "IndexError: list index out of range" happens exactly at 50% point all the time











Hey @subha-aws , thanks for sharing this bug. I have also noticed this, and will fix is ASAP

hey @subha-aws can you update to v0.1. I don't think that fixes this issue but the error trace you gave is a bit misleading I think. Need the full trace for v0.1. If you could pass that that would be really helpful for us to reproduce :)

A little more descriptive error with v0.1  
```
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/executor.py", line 75, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/nest_asyncio.py", line 99, in run_until_complete
    return f.result()
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/executor.py", line 63, in _aresults
    raise e
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/executor.py", line 58, in _aresults
    r = await future
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/executor.py", line 91, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/testset/extractor.py", line 49, in extract
    results = await self.llm.generate(prompt=prompt, is_async=is_async)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/llms/base.py", line 92, in generate
    return await agenerate_text_with_retry(
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
    return await fn(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/tenacity/_asyncio.py", line 47, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/tenacity/__init__.py", line 325, in iter
    raise retry_exc.reraise()
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/tenacity/__init__.py", line 158, in reraise
    raise self.last_attempt.result()
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/tenacity/_asyncio.py", line 50, in __call__
    result = await fn(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/llms/base.py", line 177, in agenerate_text
    result = await self.langchain_llm.agenerate_prompt(
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 540, in agenerate_prompt
    return await self.agenerate(
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 815, in agenerate
    isinstance(callbacks[0], (list, BaseCallbackManager))
IndexError: list index out of range
---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
Cell In[10], line 54
     41 docstore = InMemoryDocumentStore(
     42     splitter=splitter,
     43     embeddings=embeddings_model,
     44     extractor=KeyphraseExtractor(generator_llm),
     45 )
     47 test_generator = TestsetGenerator(
     48     generator_llm=generator_llm,
     49     critic_llm=critic_llm,
     50     embeddings=titan_embeddings,
     51     docstore=docstore
     52 )
---> 54 testset = test_generator.generate_with_langchain_docs(documents=documents,
     55                                                       test_size=10,
     56                                                       distributions=testset_distribution)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/testset/generator.py:150, in TestsetGenerator.generate_with_langchain_docs(self, documents, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
    139 def generate_with_langchain_docs(
    140     self,
    141     documents: t.Sequence[LCDocument],
   (...)
    148 ):
    149     # chunk documents and add to docstore
--> 150     self.docstore.add_documents(
    151         [Document.from_langchain_document(doc) for doc in documents]
    152     )
    154     return self.generate(
    155         test_size=test_size,
    156         distributions=distributions,
   (...)
    160         run_config=run_config,
    161     )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/testset/docstore.py:210, in InMemoryDocumentStore.add_documents(self, docs, show_progress)
    204 # split documents with self.splitter into smaller nodes
    205 nodes = [
    206     Node.from_langchain_document(d)
    207     for d in self.splitter.transform_documents(docs)
    208 ]
--> 210 self.add_nodes(nodes, show_progress=show_progress)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/testset/docstore.py:250, in InMemoryDocumentStore.add_nodes(self, nodes, show_progress, desc)
    248 results = executor.results()
    249 if results == []:
--> 250     raise ExceptionInRunner()
    252 for i, n in enumerate(nodes):
    253     if i in nodes_to_embed.keys():

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exception=False` incase you want to show only a warning message instead.
```

wow it seems like the error is coming from langchain because there are not callbacks used - very strange. could you add a callback to test it? please refer https://docs.ragas.io/en/stable/howtos/applications/tracing.html

Hey @subha-aws Is this a new error? I have raised a fix for the earlier one you posted and was able to run test gen on bedrock
```
from langchain_community.chat_models import BedrockChat
from langchain_community.embeddings import BedrockEmbeddings
from ragas.embeddings.base import BaseRagasEmbeddings, LangchainEmbeddingsWrapper
from ragas.llms import BaseRagasLLM, LangchainLLMWrapper


config = {
   your-config
}

bedrock_model = BedrockChat(
    credentials_profile_name=config["credentials_profile_name"],
    region_name=config["region_name"],
    endpoint_url=f"https://bedrock-runtime.{config['region_name']}.amazonaws.com",
    model_id=config["model_id"],
    model_kwargs=config["model_kwargs"],
)
bedrock_model = LangchainLLMWrapper(bedrock_model)
# init the embeddings
bedrock_embeddings = BedrockEmbeddings(
    credentials_profile_name=config["credentials_profile_name"],
    region_name=config["region_name"],
)

bedrock_embeddings = LangchainEmbeddingsWrapper(bedrock_embeddings)


###Add custom llms and embeddings
generator_llm = bedrock_model
critic_llm = bedrock_model

from ragas.testset.extractor import KeyphraseExtractor
from langchain.text_splitter import TokenTextSplitter
from ragas.testset.docstore import InMemoryDocumentStore


splitter = TokenTextSplitter(chunk_size=1000, chunk_overlap=100)
keyphrase_extractor = KeyphraseExtractor(llm=generator_llm)
docstore = InMemoryDocumentStore(
splitter=splitter,
embeddings=bedrock_embeddings,
extractor=keyphrase_extractor,
)
from ragas.testset import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context

test_generator = TestsetGenerator(
generator_llm=generator_llm,
critic_llm=critic_llm,
embeddings=bedrock_embeddings,
docstore=docstore,
)

testset = test_generator.generate_with_llamaindex_docs(documents=documents[:5],
test_size=10,distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})

```

Hey @subha-aws once the PR is merged can you reinstall from source and try again? 

> Hey @subha-aws once the PR is merged can you reinstall from source and try again?

Sure will test this out today. btw, I was using langchain documents not llama index. I will check them out either way!  Thank you!


cool, feel free to close this issue if it works out for you. @subha-aws 

@shahules786  , Can I add this to the documentation for Bedrock? I also have a notebook for bedrock for LLM comparison.

Yes please @subha-aws . I think it would be best to add it [here](https://docs.ragas.io/en/stable/howtos/customisations/aws-bedrock.html)
Like I have added for [language adaptation](https://docs.ragas.io/en/stable/howtos/applications/use_prompt_adaptation.html), create a section for metrics and test generation. If you need clarity, happy to hoping a [call](https://calendly.com/shahules/30min) 

Let me know if you're working on it so that I can create an issue and assign it to you and make sure non one else is doing the same. @subha-aws 

@shahules786  Yes please do. 

> cool, feel free to close this issue if it works out for you. @subha-aws

This works! Thank you

Has this issue been closed?
I had the similar error and I was using the latest version of ragas, and I used LangchainLLMWrapper.

File "/home/admin1/miniconda3/envs/lccc/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 853, in agenerate
    isinstance(callbacks[0], (list, BaseCallbackManager))
IndexError: list index out of range



@subha-aws  Same issue. Please let me know if the issue has been resolved!

@shahules786 hi, I am getting same error as well others. I am using generate_with_langchain_docs by the way. 

please see #748

I was trying to do "Synthetic test data generation" using RAGAS framework with the help of the below document.
 
"https://docs.ragas.io/en/stable/concepts/testset_generation.html"

I'm facing error. Please have a look on the below error. 

ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
Task was destroyed but it is pending!
task: <Task pending name='Task-3' coro=<as_completed.<locals>.sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\sit
e-packages\ragas\executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]>
Task was destroyed but it is pending!
task: <Task pending name='Task-5' coro=<as_completed.<locals>.sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\sit
e-packages\ragas\executor.py:38> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]>
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<as_completed.<locals>.sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\site-packages\ragas\executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]>


@sona-16 ,I have the same question,have you already solve it ?

I have a related error, I am using VertexAI with ragas, followed this documentation 
https://docs.ragas.io/en/latest/howtos/customisations/gcp-vertexai.html ,below is the traceback:
WARNING:ragas.validation:passing column names as 'ground_truths' is deprecated and will be removed in the next version, please use 'ground_truth' instead. Note that `ground_truth` should be of type string and not Sequence[string] like `ground_truths`
Evaluating:  25%
 3/12 [00:05<00:11,  1.25s/it]
WARNING:ragas.llms.output_parser:Failed to parse output. Returning None.
WARNING:ragas.llms.output_parser:Failed to parse output. Returning None.
Exception in thread Thread-355:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 96, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 84, in _aresults
    raise e
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 79, in _aresults
    r = await future
  File "/usr/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 38, in sema_coro
    return await coro
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 112, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/base.py", line 116, in ascore
    raise e
  File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/base.py", line 112, in ascore
    score = await self._ascore(row=row, callbacks=group_cm, is_async=is_async)
  File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/_answer_relevance.py", line 167, in _ascore
    return self._calculate_score(answers, row)
  File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/_answer_relevance.py", line 139, in _calculate_score
    cosine_sim = self.calculate_similarity(question, gen_questions)
  File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/_answer_relevance.py", line 113, in calculate_similarity
    question_vec = np.asarray(self.embeddings.embed_query(question)).reshape(1, -1)
  File "/usr/local/lib/python3.10/dist-packages/langchain_google_vertexai/embeddings.py", line 391, in embed_query
    return self.embed([text], 1, "RETRIEVAL_QUERY")[0]
  File "/usr/local/lib/python3.10/dist-packages/langchain_google_vertexai/embeddings.py", line 363, in embed
    embeddings.extend(t.result())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/dist-packages/langchain_google_vertexai/embeddings.py", line 194, in _get_embeddings_with_retry
    with telemetry.tool_context_manager(self._user_agent):
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/root/.local/lib/python3.10/site-packages/google/cloud/aiplatform/telemetry.py", line 48, in tool_context_manager
    _pop_tool_name(tool_name)
  File "/root/.local/lib/python3.10/site-packages/google/cloud/aiplatform/telemetry.py", line 57, in _pop_tool_name
    raise RuntimeError(
RuntimeError: Tool context error detected. This can occur due to parallelization.
---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
<ipython-input-36-4e37e43ed01c> in <cell line: 1>()
----> 1 metrices = getMetrices(dataset)

2 frames
/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py in evaluate(dataset, metrics, llm, embeddings, callbacks, is_async, run_config, raise_exceptions, column_map)
    211         results = executor.results()
    212         if results == []:
--> 213             raise ExceptionInRunner()
    214 
    215         # convert results to dataset_like

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

Please help me with this asap! @subha-aws 

 same error persists ...
 
 
 252 results = executor.results()
    253 if not results:
--> 254     raise ExceptionInRunner()
    256 for i, n in enumerate(nodes):
    257     if i in nodes_to_embed.keys():

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.