Repository: explodinggradients/ragas
Issue: Getting detailed information from metrics
Root cause: Documentation Deficiency

Is there a way to get more information rom the metrics? For instance, can we use the callbacks somehow to get the questions generated in Answer Relevancy metric? This would be great for debugging and finding out why a specific metric is failing, and if it would be better to define custom ones.


Hi! Great question about getting more detailed information from metrics for debugging purposes.

Yes, there are several ways to get detailed information from metrics, including the questions generated in Answer Relevancy:

## 1. Using the New Metric API (Individual Metric Calls)

The new Metric API returns `MetricResult` objects that include a `.traces` property with detailed input/output information:

```python
from ragas.metrics import answer_relevancy
from ragas.llms import llm_factory
from ragas.embeddings import embedding_factory

# Initialize the metric
metric = answer_relevancy
metric.llm = llm_factory()
metric.embeddings = embedding_factory()
metric.init()

# Score a single sample
result = await metric.ascore(
    row={
        "user_input": "What is the capital of France?",
        "response": "Paris is the capital of France."
    },
    callbacks=[]
)

# Access the traces to see intermediate results
print(result.traces)  # Contains 'input' and 'output' data
print(result.value)   # The actual score
print(result.reason)  # Reasoning if available
```

## 2. Using Evaluation Results with Traces

When you run `evaluate()`, the results object contains a `.traces` property that captures all the intermediate steps:

```python
from ragas import evaluate
from ragas.metrics import answer_relevancy

result = evaluate(
    dataset=your_dataset,
    metrics=[answer_relevancy]
)

# Access traces for each row
for i, trace in enumerate(result.traces):
    print(f"\n--- Row {i} ---")
    # Each trace contains metric details
    if 'answer_relevancy' in trace:
        metric_trace = trace['answer_relevancy']
        # This contains the prompt inputs/outputs for that metric
        for prompt_name, prompt_data in metric_trace.items():
            if isinstance(prompt_data, dict):
                print(f"Prompt: {prompt_name}")
                print(f"Input: {prompt_data.get('input')}")
                print(f"Output: {prompt_data.get('output')}")
```

## 3. Using Custom Callbacks for Advanced Tracing

You can also use the callback system to capture detailed information during evaluation. The built-in `RagasTracer` callback automatically captures all intermediate steps:

```python
from ragas.callbacks import RagasTracer
from ragas import evaluate

# Create a tracer
tracer = RagasTracer()

# Run evaluation with tracer
result = evaluate(
    dataset=your_dataset,
    metrics=[answer_relevancy],
    callbacks=[tracer]
)

# The tracer captures all chain runs including:
# - Evaluation level
# - Row level
# - Metric level  
# - Prompt level (this is where you'll see generated questions!)

# Access the raw traces
print(tracer.to_jsons())  # Get all traces as JSON
```

## For Answer Relevancy Specifically

The Answer Relevancy metric generates multiple questions (controlled by the `strictness` parameter, default=3) and you can access these through traces. The generated questions are in the prompt outputs:

```python
from ragas.metrics import AnswerRelevancy

# You can also customize the number of questions generated
custom_metric = AnswerRelevancy(strictness=5)  # Generate 5 questions instead of 3
```

The traces will show you:
- The input response that was evaluated
- Each generated question
- Whether each question was marked as "noncommittal"
- The cosine similarity scores between original and generated questions

## Documentation

For more details on tracing, check out the [tracing documentation](https://docs.ragas.io/en/latest/howtos/customizations/metrics/tracing/) which covers integration with observability tools like LangSmith, Wandb, and Opik.

Hope this helps! Let me know if you need any clarification on accessing specific information from the traces.

Great, that is exactly what I am looking for. Looks like I wasn't thorough enough in looking at the documentation!

glad to hear that it helped 🎉 
hope we see you around soon with more questions, feedbacks etc 🙂 ❤️