Repository: explodinggradients/ragas
Issue: Evaluation Error with Gemini/VertexAI: InvalidArgument due to Empty Text Content
Root cause: Validation Gap

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**

When attempting to evaluate a dataset using Gemini, an `InvalidArgument` error occurs indicating that the text content is empty. This issue arises specifically when using the evaluate function with the `VertexAIEmbeddings` from Langchain. Empty strings in the dataset seem to cause the error during the embedding process. The same evaluation process works without errors when using the OpenAI embeddings. The `answer_similarity` metric is used for evaluation.

Ragas version: 0.1.8
Python version: 3.10.13

**Code to Reproduce**

```python
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_similarity

data = {
    "answer": [
        "Paris is the capital of France.",
        "",
        "Paris is the capital of France.",
        "",
    ],
    "ground_truth": [
        "Paris",
        "Paris",
        "",
        "",
    ],
}
dataset = Dataset.from_dict(data)
```

With OpenAI (no error):

```python
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

llm = ChatOpenAI(model="gpt-3.5-turbo")
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")

result = evaluate(
    dataset,
    metrics=[answer_similarity],
    llm=llm,
    embeddings=embeddings,
)
result.to_pandas()
```

Output:

| answer                          | ground_truth | answer_similarity |
| ------------------------------- | ------------ | ----------------- |
| Paris is the capital of France. | Paris        | 0.493884          |
| ""                              | Paris        | 0.175841          |
| Paris is the capital of France. | ""           | 0.101094          |
| ""                              | ""           | 1.000000          |

With Gemini (error):

```python
from langchain_google_vertexai import ChatVertexAI, VertexAIEmbeddings

llm = ChatVertexAI(model_name="gemini-1.5-flash-preview-0514")
embeddings = VertexAIEmbeddings(model_name="text-embedding-004")

result = evaluate(
    dataset,
    metrics=[answer_similarity],
    llm=llm,
    embeddings=embeddings,
)
result.to_pandas()
```

**Error trace**

<details>
  <summary>Open error trace</summary>

  ```python
  Traceback (most recent call last):
    File "/Users/baptiste.pasquier/Github/ragas/src/ragas/executor.py", line 79, in _aresults
      r = await future
    File "/Users/baptiste.pasquier/.pyenv/versions/3.10.13/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
      return f.result()  # May raise f.exception().
    File "/Users/baptiste.pasquier/Github/ragas/src/ragas/executor.py", line 38, in sema_coro
      return await coro
    File "/Users/baptiste.pasquier/Github/ragas/src/ragas/executor.py", line 112, in wrapped_callable_async
      return counter, await callable(*args, **kwargs)
    File "/Users/baptiste.pasquier/Github/ragas/src/ragas/metrics/base.py", line 125, in ascore
      raise e
    File "/Users/baptiste.pasquier/Github/ragas/src/ragas/metrics/base.py", line 121, in ascore
      score = await self._ascore(row=row, callbacks=group_cm, is_async=is_async)
    File "/Users/baptiste.pasquier/Github/ragas/src/ragas/metrics/_answer_similarity.py", line 65, in _ascore
      embedding_1 = np.array(await self.embeddings.embed_text(ground_truth))
    File "/Users/baptiste.pasquier/Github/ragas/src/ragas/embeddings/base.py", line 23, in embed_text
      embs = await self.embed_texts([text], is_async=is_async)
    File "/Users/baptiste.pasquier/Github/ragas/src/ragas/embeddings/base.py", line 33, in embed_texts
      return await aembed_documents_with_retry(texts)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 142, in async_wrapped
      return await fn(*args, **kwargs)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 58, in __call__
      do = await self.iter(retry_state=retry_state)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 110, in iter
      result = await action(retry_state)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 78, in inner
      return fn(*args, **kwargs)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 410, in exc_check
      raise retry_exc.reraise()
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 183, in reraise
      raise self.last_attempt.result()
    File "/Users/baptiste.pasquier/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/_base.py", line 451, in result
      return self.__get_result()
    File "/Users/baptiste.pasquier/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
      raise self._exception
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 61, in __call__
      result = await fn(*args, **kwargs)
    File "/Users/baptiste.pasquier/Github/ragas/src/ragas/embeddings/base.py", line 64, in aembed_documents
      return await self.embeddings.aembed_documents(texts)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/langchain_core/embeddings/embeddings.py", line 21, in aembed_documents
      return await run_in_executor(None, self.embed_documents, texts)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/langchain_core/runnables/config.py", line 514, in run_in_executor
      return await asyncio.get_running_loop().run_in_executor(
    File "/Users/baptiste.pasquier/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/thread.py", line 58, in run
      result = self.fn(*self.args, **self.kwargs)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/langchain_google_vertexai/embeddings.py", line 380, in embed_documents
      return self.embed(texts, batch_size, "RETRIEVAL_DOCUMENT")
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/langchain_google_vertexai/embeddings.py", line 363, in embed
      embeddings.extend(t.result())
    File "/Users/baptiste.pasquier/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/_base.py", line 451, in result
      return self.__get_result()
    File "/Users/baptiste.pasquier/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
      raise self._exception
    File "/Users/baptiste.pasquier/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/thread.py", line 58, in run
      result = self.fn(*self.args, **self.kwargs)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/langchain_google_vertexai/embeddings.py", line 197, in _get_embeddings_with_retry
      return self._get_text_embeddings_with_retry(
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/langchain_google_vertexai/embeddings.py", line 228, in _get_text_embeddings_with_retry
      embeddings = self.instance["get_embeddings_with_retry"](requests)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 330, in wrapped_f
      return self(f, *args, **kw)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 467, in __call__
      do = self.iter(retry_state=retry_state)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 368, in iter
      result = action(retry_state)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 390, in <lambda>
      self._add_action_func(lambda rs: rs.outcome.result())
    File "/Users/baptiste.pasquier/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/_base.py", line 451, in result
      return self.__get_result()
    File "/Users/baptiste.pasquier/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
      raise self._exception
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 470, in __call__
      result = fn(*args, **kwargs)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/vertexai/language_models/_language_models.py", line 2145, in get_embeddings
      prediction_response = self._endpoint.predict(
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/google/cloud/aiplatform/models.py", line 1614, in predict
      prediction_response = self._prediction_client.predict(
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 851, in predict
      response = rpc(
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
      return wrapped_func(*args, **kwargs)
    File "/Users/baptiste.pasquier/Github/ragas/.venv/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
      raise exceptions.from_grpc_error(exc) from exc
  google.api_core.exceptions.InvalidArgument: 400 The text content is empty.
  ```

</details>


Here is the solution to this:

```
contexts = []
answers = []
import time

# Generate contexts and answers for each question
for query in questions:
    result = qa({"query": query})
    time.sleep(20) # uncomment for quota issues
    contexts.append(
        [document.page_content for document in result.get("source_documents")]
    )
    answers.append(result.get("result"))

# Convert into a dataset and prepare for consumtion by RAGAS API
data = {
    "question": questions,
    "contexts": contexts,
    "ground_truth": ground_truth,
    "answer": answers,
}

dataset = Dataset.from_dict(data)
dataset

# Compile list of RAGAS Metrics
metrics = [
    answer_relevancy,
    context_recall,
    context_precision,
    harmfulness,
    answer_similarity,
]

## IMPORTANT : Gemini with RAGAS
> RAGAS is designed to work with OpenAI Models by default. We must set a few attributes to make it work with Gemini

class RAGASVertexAIEmbeddings(VertexAIEmbeddings):
    """Wrapper for RAGAS"""

    async def embed_text(self, text: str) -> list[float]:
        """Embeds a text for semantics similarity"""
        return self.embed([text], 1, "SEMANTIC_SIMILARITY")[0]

# Wrapper to make RAGAS work with Gemini and Vertex AI Embeddings Models
embeddings = RAGASVertexAIEmbeddings(model_name="textembedding-gecko@003")
ragas_llm = LangchainLLMWrapper(llm)

for m in metrics:
    # change LLM for metric
    m.__setattr__("llm", ragas_llm)

    # check if this metric needs embeddings
    if hasattr(m, "embeddings"):
        # if so change with Vertex AI Embeddings
        m.__setattr__("embeddings", embeddings)

### Run the RAGAS Evaluation

# Run the evaluation on every row of the dataset
result_set = []
for i in range(len(dataset)):
    result = evaluate(
        dataset=Dataset.from_dict(dataset[i : i + 1]),
        metrics=metrics,
        raise_exceptions=False,
    )
    result_set.append(result.to_pandas())
```

Thanks to @adi885 and @holtskinner

if you need complete working code, here it is:  https://github.com/anshupandey/generative-ai/blob/main/gemini/use-cases/retrieval-augmented-generation/rag-evaluation/ragas_with_gemini.ipynb

(As the code by Google has basic authentication part missing and installation errors).

thanks a lot @anshupandey for helping ❤️ 🙌🏽 

It seems the issue was answered, closing this now.