Repository: explodinggradients/ragas
Issue: RAGAS without ground_truth
Root cause: Documentation Deficiency

Hi, is it possible to use the library without having the ground_truth for any metrics?


<!-- Greeting -->
Hey @dasilvaviviane! I'm here to help you with any issues or questions you have while waiting for a human maintainer. I'm a bot 🤖 that can assist you with bug fixes, answer questions, and guide you on how to become a contributor.

<!-- Answer -->
Yes, it is possible to use the RAGAS library without having the ground_truth for any metrics. Ragas can perform reference-free evaluations of various aspects of your RAG pipeline, meaning you don't need ground-truths when running the evaluations and can run it on production traces that you've collected with Langfuse. The library supports both LLM-based and non-LLM-based metrics, allowing for flexible evaluation methods that do not necessarily require ground-truth data <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/langfuse.ipynb)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/metrics/index.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


how to evaluate ragas metrics if any key user input or contexts or answer or reference is not present or null
@dosu


you can checkout the General Purpose Metrics mentioned in the docs too - https://docs.ragas.io/en/latest/concepts/metrics (bottom)

<!-- Answer -->
If any key user input, context, answer, or reference is not present or is null, the `_single_turn_ascore` method will not be able to generate the necessary statements for evaluation, and the `_ascore` method will return `np.nan` as the score. Additionally, the `evaluate` function in the RAGAS framework will raise a `ValueError` if the dataset is not provided or if the columns required for the metrics are missing or of the wrong format.

```python
if answers is None:
    return np.nan

if dataset is None:
    raise ValueError("Provide dataset!")
```

The `evaluate` function also includes a `raise_exceptions` parameter, which, if set to `False`, will return `np.nan` for the row that failed instead of raising an exception.

```python
raise_exceptions: bool = False,
```

This ensures that the evaluation process can handle missing or null values gracefully by either returning `np.nan` or raising an appropriate exception based on the configuration <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_correctness.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/evaluation.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Thanks @jjmachan for quite good link that you posted.
Also I have a scenario where my code is hosted or there is app running. I have golden dataset by client that has 2 columns question and answer only. I have to get the answer from LLM and contexts retrieved at runtime for the same question that client gave. Then the metrics from RAGAS can be evaluated. How to carry out this. Any idea? 
@dosu

Also like what is the stable version of RAGAS as from pypi we have 0.1.20, but in this version many functions will be depreciated.

> you can checkout the General Purpose Metrics mentioned in the docs too - https://docs.ragas.io/en/latest/concepts/metrics (bottom)

Thanks, from your suggestion, I tried to use https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/answer_relevance/#response-relevancy but I couldn't install SingleTurnSample, do you have any idea on why?

<!-- Answer -->
To evaluate RAGAS metrics using a golden dataset with only question and answer columns, while retrieving answers from an LLM and contexts at runtime, follow these steps:

1. **Prepare the Dataset**: Ensure your dataset has a "question" column. The "answer" column will be used for comparison if needed.

2. **Set Up the Query Engine**: Use a query engine to process the questions and retrieve answers and contexts at runtime. Tools like LlamaIndex can help build your RAG pipeline.

3. **Execute Queries**: Submit each question to the query engine, retrieve the answers and contexts, and construct a new dataset with these results.

4. **Evaluate with RAGAS**: Use the new dataset, which includes questions, retrieved answers, and contexts, to evaluate using RAGAS metrics.

Here's a code snippet to guide you:

```python
from datasets import Dataset
from ragas.evaluation import evaluate as ragas_evaluate
from ragas.executor import Executor

def evaluate_ragas(query_engine, dataset, metrics, llm=None, embeddings=None, raise_exceptions=False):
    if dataset is None:
        raise ValueError("Provide dataset!")

    exec = Executor(
        desc="Running Query Engine",
        keep_progress_bar=True,
        raise_exceptions=raise_exceptions,
    )

    queries = dataset["question"]
    for i, q in enumerate(queries):
        exec.submit(query_engine.aquery, q, name=f"query-{i}")

    answers = []
    contexts = []
    try:
        results = exec.results()
        if results == []:
            raise Exception("No results returned")
    except Exception as e:
        raise e
    else:
        for r in results:
            answers.append(r.response)
            contexts.append([n.node.text for n in r.source_nodes])

    hf_dataset = Dataset.from_dict(
        {
            "question": queries,
            "contexts": contexts,
            "answer": answers,
        }
    )

    results = ragas_evaluate(
        dataset=hf_dataset,
        metrics=metrics,
        llm=llm,
        embeddings=embeddings,
        raise_exceptions=raise_exceptions,
    )

    return results
```

This function will handle the evaluation process, leveraging tools like LlamaIndex for building the query engine and RAGAS for evaluation <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/integrations/llama_index.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@veenuvinod the metrics useful would be
- answer_correctness
- context_recall

others are also present but these would help you get started

@dasilvaviviane u're using the latest version - try the stable version instead

Hi How to make verbose as True so that we can see how ragas actually evaluate the metrics? That will be more helpful to understand the logic behind the metrics, in case of ascore also if we can see whats happening inside ragas.
@jjmachan @dosu

we have something like that here https://docs.ragas.io/en/stable/howtos/applications/_metrics_llm_calls/#export-llm-traces @veenuvinod. Could you check it out and let us know if that is what you where looking for?

<!-- Answer -->
To set verbose to True in RAGAS and see how the metrics are evaluated, you can call the `evaluate` function with the `verbose` parameter set to `True`. Here's an example:

```python
results = evaluate(
    dataset_name="MyDataset",
    llm_or_chain_factory=my_llm,
    experiment_name="experiment_1_with_vanila_rag",
    verbose=True
)
print(results)
```

Setting `verbose=True` will print detailed progress and results during the evaluation process, helping you understand the logic behind the metrics, including the ascore <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/integrations/langsmith.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*