Repository: explodinggradients/ragas
Issue: error: answer_relevancy with Bedrock model
Root cause: Validation Gap

**Describe the bug**
I am getting an error when using Bedrock embedding model with answer relevancy metric.

Ragas version: 0.0.21
Python version: 3.10.8

**Code to Reproduce**
from ragas import evaluate
from ragas.metrics import answer_relevancy
import nest_asyncio  # CHECK NOTES

# NOTES: Only used when running on a jupyter notebook, otherwise comment or remove this function.
nest_asyncio.apply()

result = evaluate(
    amnesty_qa["eval"].select(range(3)),
    metrics=[answer_relevancy],
    llm=bedrock_model,
    embeddings=bedrock_embeddings,
)

result

**Error trace**
---------------------------------------------------------------------------
ValidationException                       Traceback (most recent call last)
File [/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:131](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:131), in BedrockEmbeddings._embedding_func(self, text)
    [129](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:129) try:
    [130](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:130)     # invoke bedrock API
--> [131](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:131)     response = self.client.invoke_model(
    [132](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:132)         body=body,
    [133](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:133)         modelId=self.model_id,
    [134](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:134)         accept="application/json",
    [135](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:135)         contentType="application/json",
    [136](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:136)     )
    [138](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:138)     # format output based on provider

File [/usr/local/lib/python3.10/site-packages/botocore/client.py:553](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/botocore/client.py:553), in ClientCreator._create_api_method.<locals>._api_call(self, *args, **kwargs)
    [552](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/botocore/client.py:552) # The "self" in this scope is referring to the BaseClient.
--> [553](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/botocore/client.py:553) return self._make_api_call(operation_name, kwargs)

File [/usr/local/lib/python3.10/site-packages/botocore/client.py:1009](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/botocore/client.py:1009), in BaseClient._make_api_call(self, operation_name, api_params)
   [1008](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/botocore/client.py:1008)     error_class = self.exceptions.from_code(error_code)
-> [1009](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/botocore/client.py:1009)     raise error_class(parsed_response, operation_name)
   [1010](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/botocore/client.py:1010) else:

ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: expected minLength: 1, actual: 0, please reformat your input and try again.

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[42], [line 1](vscode-notebook-cell:?execution_count=42&line=1)
----> [1](vscode-notebook-cell:?execution_count=42&line=1) result = evaluate(
      [2](vscode-notebook-cell:?execution_count=42&line=2)     amnesty_qa["eval"].select(range(3)),
      [3](vscode-notebook-cell:?execution_count=42&line=3)     metrics=[answer_relevancy],
      [4](vscode-notebook-cell:?execution_count=42&line=4)     llm=bedrock_model,
      [5](vscode-notebook-cell:?execution_count=42&line=5)     embeddings=bedrock_embeddings,
      [6](vscode-notebook-cell:?execution_count=42&line=6) )

File [~/Projects/ragas/src/ragas/evaluation.py:204](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:204), in evaluate(dataset, metrics, llm, embeddings, callbacks, is_async, max_workers, raise_exceptions, column_map)
    [201](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:201)     if not evaluation_group_cm.ended:
    [202](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:202)         evaluation_rm.on_chain_error(e)
--> [204](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:204)     raise e
    [205](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:205) else:
    [206](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:206)     result = Result(
    [207](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:207)         scores=Dataset.from_list(scores),
    [208](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:208)         dataset=dataset,
    [209](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:209)         binary_columns=binary_metrics,
    [210](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:210)     )

File [~/Projects/ragas/src/ragas/evaluation.py:187](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:187), in evaluate(dataset, metrics, llm, embeddings, callbacks, is_async, max_workers, raise_exceptions, column_map)
    [184](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:184) scores = []
    [185](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:185) try:
    [186](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:186)     # get the results
--> [187](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:187)     results = executor.results()
    [188](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:188)     # convert results to dataset_like
    [189](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/evaluation.py:189)     for i, _ in enumerate(dataset):

File [~/Projects/ragas/src/ragas/executor.py:119](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:119), in Executor.results(self)
    [117](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:117)     r = (-1, np.nan)
    [118](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:118)     if self.raise_exceptions:
--> [119](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:119)         raise e
    [120](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:120) finally:
    [121](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:121)     results.append(r)

File [~/Projects/ragas/src/ragas/executor.py:115](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:115), in Executor.results(self)
    [113](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:113) r = (-1, np.nan)
    [114](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:114) try:
--> [115](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:115)     r = future.result()
    [116](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:116) except Exception as e:
    [117](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:117)     r = (-1, np.nan)

File [/usr/local/Cellar/python](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python)@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py:451, in Future.result(self, timeout)
    [449](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py:449)     raise CancelledError()
    [450](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py:450) elif self._state == FINISHED:
--> [451](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py:451)     return self.__get_result()
    [453](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py:453) self._condition.wait(timeout)
    [455](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py:455) if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File [/usr/local/Cellar/python](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python)@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py:403, in Future.__get_result(self)
    [401](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py:401) if self._exception:
    [402](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py:402)     try:
--> [403](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py:403)         raise self._exception
    [404](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py:404)     finally:
    [405](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py:405)         # Break a reference cycle with the exception in self._exception
    [406](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py:406)         self = None

File [/usr/local/Cellar/python](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python)@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py:58, in _WorkItem.run(self)
     [55](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py:55)     return
     [57](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py:57) try:
---> [58](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py:58)     result = self.fn(*self.args, **self.kwargs)
     [59](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py:59) except BaseException as exc:
     [60](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py:60)     self.future.set_exception(exc)

File [~/Projects/ragas/src/ragas/executor.py:37](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:37), in Executor.wrap_callable_with_index.<locals>.wrapped_callable(*args, **kwargs)
     [36](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:36) def wrapped_callable(*args, **kwargs):
---> [37](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/executor.py:37)     return counter, callable(*args, **kwargs)

File [~/Projects/ragas/src/ragas/metrics/base.py:76](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:76), in Metric.score(self, row, callbacks)
     [74](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:74)     if not group_cm.ended:
     [75](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:75)         rm.on_chain_error(e)
---> [76](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:76)     raise e
     [77](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:77) else:
     [78](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:78)     if not group_cm.ended:

File [~/Projects/ragas/src/ragas/metrics/base.py:72](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:72), in Metric.score(self, row, callbacks)
     [68](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:68) rm, group_cm = new_group(
     [69](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:69)     self.name, inputs=row, callbacks=callbacks, is_async=False
     [70](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:70) )
     [71](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:71) try:
---> [72](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:72)     score = self._score(row=row, callbacks=group_cm)
     [73](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:73) except Exception as e:
     [74](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/base.py:74)     if not group_cm.ended:

File [~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:133](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:133), in AnswerRelevancy._score(self, row, callbacks)
    [124](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:124) result = self.llm.generate_text(
    [125](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:125)     prompt,
    [126](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:126)     n=self.strictness,
    [127](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:127)     callbacks=callbacks,
    [128](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:128) )
    [129](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:129) response = [
    [130](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:130)     json_loader.safe_load(r.text, self.llm) for r in result.generations[0]
    [131](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:131) ]
--> [133](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:133) return self._calculate_score(response, row)

File [~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:111](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:111), in AnswerRelevancy._calculate_score(self, response, row)
    [101](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:101) gen_questions = [
    [102](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:102)     item.get("question", "") for item in response if isinstance(item, dict)
    [103](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:103) ]
    [104](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:104) committal = np.any(
    [105](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:105)     [
    [106](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:106)         item.get("noncommittal", False)
   (...)
    [109](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:109)     ]
    [110](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:110) )
--> [111](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:111) cosine_sim = self.calculate_similarity(question, gen_questions)
    [112](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:112) score = cosine_sim.mean() * int(not committal)
    [114](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:114) return score

File [~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:87](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:87), in AnswerRelevancy.calculate_similarity(self, question, generated_questions)
     [84](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:84) assert self.embeddings is not None
     [85](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:85) question_vec = np.asarray(self.embeddings.embed_query(question)).reshape(1, -1)
     [86](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:86) gen_question_vec = np.asarray(
---> [87](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:87)     self.embeddings.embed_documents(generated_questions)
     [88](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:88) )
     [89](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:89) norm = np.linalg.norm(gen_question_vec, axis=1) * np.linalg.norm(
     [90](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:90)     question_vec, axis=1
     [91](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:91) )
     [92](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:92) return (
     [93](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:93)     np.dot(gen_question_vec, question_vec.T).reshape(
     [94](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:94)         -1,
     [95](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:95)     )
     [96](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:96)     [/](https://file+.vscode-resource.vscode-cdn.net/) norm
     [97](https://file+.vscode-resource.vscode-cdn.net/Users/tinomaxthayil/Projects/ragas/docs/howtos/customisations/~/Projects/ragas/src/ragas/metrics/_answer_relevance.py:97) )

File [/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:159](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:159), in BedrockEmbeddings.embed_documents(self, texts)
    [157](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:157) results = []
    [158](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:158) for text in texts:
--> [159](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:159)     response = self._embedding_func(text)
    [160](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:160)     results.append(response)
    [161](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:161) return results

File [/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:146](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:146), in BedrockEmbeddings._embedding_func(self, text)
    [144](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:144)         return response_body.get("embedding")
    [145](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:145) except Exception as e:
--> [146](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.10/site-packages/langchain_community/embeddings/bedrock.py:146)     raise ValueError(f"Error raised by inference endpoint: {e}")

ValueError: Error raised by inference endpoint: An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: expected minLength: 1, actual: 0, please reformat your input and try again.



the same here for 
Python 3.11.4
ragas 0.0.22

I'm working with bedrock claude model as well. Can you please share what embedding model you used here?
One things i dont really understand is if we using bedrock model, then we have to use embedding model from bedrock?
Should this not be matching the embedding model actually used for inference?


And also did you obeserved that with bedrock ofthen the metric result  i obtained will be "null", "nan" for faithfullness and context_precision. 
"context_precision\":null,\"faithfulness\":null,\"answer_relevancy\":0.7777600149

hey @shahules786 made some improvements here, could you take a look?