Repository: explodinggradients/ragas
Issue: ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead
Root cause: External Dependency Breakage

I am using ragas to evaluate my rag pipeline. I am getting error while running evaluate function. Here is the full error:

WARNING:ragas.validation:passing column names as 'ground_truths' is deprecated and will be removed in the next version, please use 'ground_truth' instead. Note that `ground_truth` should be of type string and not Sequence[string] like `ground_truths`
Evaluating:  42%
 5/12 [00:05<00:03,  1.81it/s]
WARNING:ragas.llms.output_parser:Failed to parse output. Returning None.
WARNING:ragas.llms.output_parser:Failed to parse output. Returning None.
WARNING:ragas.llms.output_parser:Failed to parse output. Returning None.
WARNING:ragas.llms.output_parser:Failed to parse output. Returning None.
Exception in thread Thread-267:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 96, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 84, in _aresults
    raise e
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 79, in _aresults
    r = await future
  File "/usr/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 38, in sema_coro
    return await coro
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 112, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/base.py", line 116, in ascore
    raise e
  File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/base.py", line 112, in ascore
    score = await self._ascore(row=row, callbacks=group_cm, is_async=is_async)
  File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/_answer_relevance.py", line 167, in _ascore
    return self._calculate_score(answers, row)
  File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/_answer_relevance.py", line 139, in _calculate_score
    cosine_sim = self.calculate_similarity(question, gen_questions)
  File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/_answer_relevance.py", line 113, in calculate_similarity
    question_vec = np.asarray(self.embeddings.embed_query(question)).reshape(1, -1)
  File "/usr/local/lib/python3.10/dist-packages/langchain_google_vertexai/embeddings.py", line 391, in embed_query
    return self.embed([text], 1, "RETRIEVAL_QUERY")[0]
  File "/usr/local/lib/python3.10/dist-packages/langchain_google_vertexai/embeddings.py", line 363, in embed
    embeddings.extend(t.result())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/dist-packages/langchain_google_vertexai/embeddings.py", line 194, in _get_embeddings_with_retry
    with telemetry.tool_context_manager(self._user_agent):
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/root/.local/lib/python3.10/site-packages/google/cloud/aiplatform/telemetry.py", line 48, in tool_context_manager
    _pop_tool_name(tool_name)
  File "/root/.local/lib/python3.10/site-packages/google/cloud/aiplatform/telemetry.py", line 57, in _pop_tool_name
    raise RuntimeError(
RuntimeError: Tool context error detected. This can occur due to parallelization.
---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
[<ipython-input-34-4e37e43ed01c>](https://localhost:8080/#) in <cell line: 1>()
----> 1 metrices = getMetrices(dataset)

2 frames
[/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py](https://localhost:8080/#) in evaluate(dataset, metrics, llm, embeddings, callbacks, is_async, run_config, raise_exceptions, column_map)
    211         results = executor.results()
    212         if results == []:
--> 213             raise ExceptionInRunner()
    214 
    215         # convert results to dataset_like

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

This is the code i used:

metrics = [
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
  ]
 
  config = {
    "project_id": "my-project-id",
  }

  creds, _ = google.auth.default(quota_project_id="my-project-id")

  llm = ChatVertexAI(model_name="chat-bison@002", credentials=creds)
  ragas_vertexai_llm = LangchainLLMWrapper(llm)
  vertexai_embeddings = VertexAIEmbeddings(model_name="textembedding-gecko@003", credentials=creds)
  for m in metrics:
    # change LLM for metric
    m.__setattr__("llm", ragas_vertexai_llm)

    # check if this metric needs embeddings
    if hasattr(m, "embeddings"):
        # if so change with VertexAI Embeddings
        m.__setattr__("embeddings", vertexai_embeddings)


  result = evaluate(
    dataset,  # using 1 as example due to quota constrains
    metrics=metrics,
  )

please help me out on this asap!

@sadaf0714  I am having the same issue. How did you fix it?

Any update on this ?