Repository: explodinggradients/ragas
Issue: implementing ragas
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
what is unclear to you? What would you like to know?
I have a few problems I would appreciate if you could clarify it for me.
1. Should the dataset for running evaluation only be in "str"? I have the embeddings and created the dataset just in the same format that exist in the website. but when I run it for evaluation, it tells me it should be in "str". what can I do if I want to use the embeddings instead of "str"

**Code Examples**
{
'question' =
[[-0.019296993, -0.005491396, ],....]
'answer'=[[-0.019296993, -0.005491396, 0.019954083, -0.015649473, -0.016722273, 0.010982792, -0.015019203, ,...]]
contexts' =[ [-0.019296993],[-0.005491396],[-0.015649473]]
[[[...], [...],

}
**Additional context**
for evaluating my rag do I need to set the LLM also? is it mandatory?
please help for better understanding
is it mandatory to connect to Openai?


<!-- Greeting -->
Hey @amin-kh96! I'm here to help you with any bugs, questions, or contributions. Let's tackle this together!

<!-- Answer -->
To use embeddings instead of "str" for the dataset when running evaluation in Ragas, you can pass the `embeddings` parameter to the `evaluate` function. It is not mandatory to set the LLM and connect to OpenAI for evaluation unless the metrics you are using require an LLM.

Here is an example of how to use embeddings:

```python
from ragas import evaluate
from ragas.embeddings.base import embedding_factory

# Assuming you have your dataset ready
dataset = ...  # Your dataset here

# Create embeddings
embeddings = embedding_factory()

# Run evaluation with embeddings
result = evaluate(dataset, embeddings=embeddings)
print(result)
```

The `embeddings` parameter is optional, and if not provided, Ragas will use the default embeddings for metrics that require them. Similarly, the `llm` parameter is optional and only needed if the metrics require an LLM.

If you want to use custom embeddings or LLMs, you can pass them directly to the `evaluate` function. For example, using Langchain embeddings and LLMs:

```python
from langchain_together import Together
from langchain_together.embeddings import TogetherEmbeddings

together_key = "<your-key-here>"

embeddings = TogetherEmbeddings(model="togethercomputer/m2-bert-80M-8k-retrieval")

together_completion = Together(
    model="NousResearch/Nous-Hermes-2-Mixtral-8x7B-SFT",
    temperature=0.7,
    max_tokens=4000,
    top_k=1,
    together_api_key=together_key
)

from ragas.metrics import faithfullness
from ragas import evaluate

results = evaluate(metrics=[faithfullness], llm=together_completion, embeddings=embeddings)
print(results)
```

You can also attach the LLM and embeddings to specific metrics if needed:

```python
from ragas.metrics import answer_relevancy 

answer_relevancy.llm = together_completion
answer_relevancy.embeddings = embeddings

result = evaluate(metrics=[answer_relevancy])
print(result)
```

A note on precedence: LLMs and embeddings attached to metrics have higher precedence than those passed to the `evaluate()` function. This allows you to override specific metrics with different models that perform better for the metric in question <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/evaluation.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/customisations/bring-your-own-llm-or-embs.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I think we do not understand each other very well, i have an array with float values that represent the embeddings, but when I want to evaluate my dataset it gives me the error that your value like ground_truth should be str. Is it possible to compute the metrics, directly with my embedding vectors? please help me to pass this problem.

<!-- Answer -->
Yes, it is possible to compute the metrics directly with your embedding vectors instead of using string values for `ground_truth` in Ragas. Here is how you can do it:

1. **Initialize and Use Metrics with Embeddings**:
   You can initialize the metrics to use embeddings by wrapping your embeddings with `LangchainEmbeddingsWrapper`.

   ```python
   from langchain_openai.embeddings import OpenAIEmbeddings
   from ragas.embeddings import LangchainEmbeddingsWrapper
   from ragas import evaluate

   # Initialize your embeddings
   emb = OpenAIEmbeddings()
   wrapped_embeddings = LangchainEmbeddingsWrapper(emb)

   # Assuming you have your dataset ready
   dataset = ...  # Your dataset here

   # Run evaluation with embeddings
   result = evaluate(dataset, embeddings=wrapped_embeddings)
   print(result)
   ```

2. **Using Specific Metrics with Embeddings**:
   For example, the `AnswerSimilarity` metric can compute the semantic similarity between the ground truth and the generated answer using embeddings.

   ```python
   from ragas.metrics._answer_similarity import AnswerSimilarity

   answer_similarity = AnswerSimilarity()

   async def compute_similarity(sample):
       score = await answer_similarity._single_turn_ascore(sample, callbacks=None)
       return score
   ```

3. **Custom LLMs and Embeddings**:
   You can also pass any instance of `BaseRagasLLM` or `BaseRagasEmbeddings` directly to the `evaluate()` function or attach them to specific metrics.

   ```python
   from langchain_core.language_models import BaseLanguageModel
   from langchain_core.embeddings import Embeddings

   langchain_llm =  # any langchain LLM instance
   langchain_embeddings = # any langchain Embeddings instance

   results = evaluate(metrics=[], llm=langchain_llm, embeddings=langchain_embeddings)
   ```

   Or attach them to specific metrics:

   ```python
   from ragas.metrics import answer_relevancy 

   answer_relevancy.llm = langchain_llm
   answer_relevancy.embeddings = langchain_embeddings

   result = evaluate(metrics=[answer_relevancy])
   ```

This setup allows you to use embeddings for metrics computation, bypassing the need for string values for `ground_truth` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/langfuse.ipynb)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_similarity.py)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/customisations/bring-your-own-llm-or-embs.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Is it acceptable to use this format when evaluating ragas?
attention: I collect the question and answer and the context related to one topic in a dict, so I have a list of dicts
data_set = [ { 'question' : [ ],
'answer' : [ ]
'context' : [ ]
}, {... },{...} ]

It is a list of dictionaries that in each dic I have question, answer, and context with the corresponding value.


@dosu
why do we use embeddings in RAGAS?
is it important to have the embeddings?
What if we don't have the embeddings and only have textual data?
which metrics use embeddings internally?



<!-- Answer -->
In Ragas, the format you described for your dataset is not directly compatible with the expected input format. Ragas expects the dataset to be structured using classes like `SingleTurnSample` or `MultiTurnSample` when using textual data. These classes include attributes such as `user_input`, `retrieved_contexts`, `reference_contexts`, `response`, and `reference` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/dataset_schema.py)</sup>.

Regarding the use of embeddings in Ragas:

1. **Why Use Embeddings**: Embeddings are used in Ragas to compute certain metrics that require semantic understanding beyond simple string comparison. They help in evaluating the semantic similarity between different pieces of text, which is crucial for some metrics.

2. **Importance of Embeddings**: While embeddings are important for certain metrics, they are not mandatory for all evaluations. If you only have textual data, you can still perform evaluations using metrics that do not require embeddings.

3. **Metrics Using Embeddings**: Metrics that require embeddings are instances of the `MetricWithEmbeddings` class. The specific metrics are not listed in the provided context, but any metric that is a subclass of `MetricWithEmbeddings` would require embeddings for its computation <sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/evaluation.py)</sup>.

If you only have textual data, you can still evaluate your RAG pipeline using non-embedding metrics like `BleuScore` or LLM-based metrics like `AspectCritic` <sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/evals.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*

---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/b956f5d0-6b4f-4ac6-ae7f-6129231bc735?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/b956f5d0-6b4f-4ac6-ae7f-6129231bc735?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/b956f5d0-6b4f-4ac6-ae7f-6129231bc735?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/b956f5d0-6b4f-4ac6-ae7f-6129231bc735?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/b956f5d0-6b4f-4ac6-ae7f-6129231bc735?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/b956f5d0-6b4f-4ac6-ae7f-6129231bc735?feedback_type=other)</sup>


@dosu
The thing is that I have the embeddings and the textual data. but I do not know which metric to choose if I do not want to use the embeddings. and of course, I want to avoid metrics based on openAI which generates embeddings for computing similarity. give me the names, please. I need more detail


<!-- Answer -->
If you want to avoid using embeddings and OpenAI-generated embeddings for computing similarity in Ragas, you can use the following metrics:

1. **ExactMatch**: Checks if the response is exactly the same as the reference text.
2. **StringPresence**: Checks if the response contains the reference text.
3. **NonLLMStringSimilarity**: Measures similarity using traditional string distance measures like Levenshtein, Hamming, Jaro, and Jaro-Winkler <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_string.py)</sup>.

Additionally, you can consider other non-embedding, non-LLM metrics such as:

- **BLEU Score**: Evaluates the quality of a response using n-gram precision and brevity penalty.
- **ROUGE Score**: Measures overlap between generated responses and reference text based on n-gram recall, precision, and F1 score <sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/metrics/available_metrics/traditional.md)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/metrics/available_metrics/index.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*

---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/e0016f5e-774d-464b-b3a4-046cf868d9bd?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/e0016f5e-774d-464b-b3a4-046cf868d9bd?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/e0016f5e-774d-464b-b3a4-046cf868d9bd?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/e0016f5e-774d-464b-b3a4-046cf868d9bd?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/e0016f5e-774d-464b-b3a4-046cf868d9bd?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/e0016f5e-774d-464b-b3a4-046cf868d9bd?feedback_type=other)</sup>


@dosu 
no no let's make it clear
give me the name of metrics that are LLM-based and use or generate embeddings internally from API calls.
and also give me the names of metrics that are LLM-based and don't use embeddings or generate it

<!-- Answer -->
The LLM-based metrics in Ragas that use or generate embeddings internally from API calls are:

- `faithfulness`
- `answer_relevancy`
- `context_precision`

The LLM-based metric that does not use or generate embeddings is:

- `aspect_critique` (specifically `harmfulness`) <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/_langfuse.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*

---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/1a135b29-bf70-467a-b352-9abf0a6472a2?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/1a135b29-bf70-467a-b352-9abf0a6472a2?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/1a135b29-bf70-467a-b352-9abf0a6472a2?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/1a135b29-bf70-467a-b352-9abf0a6472a2?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/1a135b29-bf70-467a-b352-9abf0a6472a2?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/1a135b29-bf70-467a-b352-9abf0a6472a2?feedback_type=other)</sup>


@dosu 
Do these metrics still exist in the latest version of Ragas?
faithfulness
answer_relevancy
context_precision

<!-- Answer -->
Yes, the metrics "faithfulness," "answer_relevancy," and "context_precision" still exist in the latest version of Ragas. You can import and use these metrics from the Ragas library as shown in the following code example:

```python
from ragas.metrics import faithfulness, answer_relevancy, context_precision
```

These metrics are used to evaluate the performance of RAG pipelines <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/_langfuse.md)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/langfuse.ipynb)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*

---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/0d83d0d3-5575-4f01-b523-ff3bdfb14c47?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/0d83d0d3-5575-4f01-b523-ff3bdfb14c47?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/0d83d0d3-5575-4f01-b523-ff3bdfb14c47?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/0d83d0d3-5575-4f01-b523-ff3bdfb14c47?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/0d83d0d3-5575-4f01-b523-ff3bdfb14c47?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/0d83d0d3-5575-4f01-b523-ff3bdfb14c47?feedback_type=other)</sup>


@dosu
Does Ragas only evaluate the performance of 
 RAG pipelines?

<!-- Answer -->
No, Ragas does not only evaluate the performance of RAG pipelines. While it is used for this purpose, it also provides tools for generating synthetic test datasets to evaluate various AI applications. Ragas can be adapted to evaluate other aspects, such as domain-specific criteria or custom metrics like "SportsRelevanceMetric" and "AnswerQualityMetric." It also supports evaluating agent workflows, demonstrating its versatility beyond just RAG pipelines <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/haystack.md)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/test_data_generation/index.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*

---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/2a9af2f7-5852-4e55-9a3b-8a47654f1758?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/2a9af2f7-5852-4e55-9a3b-8a47654f1758?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/2a9af2f7-5852-4e55-9a3b-8a47654f1758?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/2a9af2f7-5852-4e55-9a3b-8a47654f1758?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/2a9af2f7-5852-4e55-9a3b-8a47654f1758?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/2a9af2f7-5852-4e55-9a3b-8a47654f1758?feedback_type=other)</sup>


@dosu
what do you mean by AI applications?
also explain this:
It also supports evaluating agent workflows, demonstrating its versatility beyond just RAG pipelines


<!-- Answer -->
In the context of Ragas, "AI applications" refers to systems or solutions that utilize artificial intelligence, including but not limited to Retrieval Augmented Generation (RAG) pipelines and agent workflows. Ragas supports evaluating agent workflows by providing tools to generate synthetic test datasets and offering a comprehensive evaluation framework. This framework allows for the assessment of key performance metrics such as tool call accuracy and agent goal accuracy. This capability highlights Ragas's versatility beyond just RAG pipelines, as it addresses the challenges of evaluating complex agentic workflows that involve multiple steps and interactions. The framework is designed to help curate a comprehensive test suite that covers various scenarios and edge cases, ensuring the effectiveness, accuracy, and responsiveness of AI agents in different domains <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/test_data_generation/index.md)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/test_data_generation/agents.md)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/langgraph_agent_evaluation.ipynb)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*

---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/6139542c-0ac1-4f9a-8479-ed2b5198eaed?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/6139542c-0ac1-4f9a-8479-ed2b5198eaed?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/6139542c-0ac1-4f9a-8479-ed2b5198eaed?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/6139542c-0ac1-4f9a-8479-ed2b5198eaed?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/6139542c-0ac1-4f9a-8479-ed2b5198eaed?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/6139542c-0ac1-4f9a-8479-ed2b5198eaed?feedback_type=other)</sup>