Repository: confident-ai/deepeval
Issue: [Feature Proposal] Insight Generation from `EvaluationResult`

Hi team! I've been using DeepEval and finding it very powerful.

After running evaluations, I often find myself manually analyzing which metrics failed and why. This gave me an idea: **a utility function that generates a short, natural-language summary of evaluation results using an LLM**.

I've built a proof-of-concept and would love to contribute this feature if you're interested.

## Motivation

When `deepeval.evaluate()` returns results, users typically need to:
1. Manually scan through `test_results` to find failures
2. Identify patterns (e.g., "Summarization failed on all 10 testcases")
3. Determine if failures are **systematic** (config issue) vs **sporadic** (model issue)

An automated insight generator could save time and surface actionable recommendations.

## Proposed API

```python
from deepeval.evaluate import evaluate
from deepeval.insights import generate_insights  # New module

results = evaluate(test_cases, metrics)

# Optional: generate insights from results
insights = generate_insights(
    evaluation_result=results,
    model=my_llm,  # DeepEvalBaseLLM instance
)

print(insights.summary)
# "Your RAG pipeline performs well on core metrics (Faithfulness: 100%, 
#  Answer Relevancy: 100%). However, Summarization failed on all 10 testcases, 
#  suggesting a configuration mismatch rather than a model issue."

print(insights.recommendations)
# [Recommendation(priority=1, action="Remove Summarization metric...", ...)]
```

## Implementation Approach

### 1. New function: `generate_insights()`

Location: `deepeval/insights/insights.py` (new module)

```python
from deepeval.evaluate.types import EvaluationResult
from deepeval.models import DeepEvalBaseLLM

def generate_insights(
    evaluation_result: EvaluationResult,
    model: DeepEvalBaseLLM,
) -> InsightResult:
    """
    Generate actionable insights from evaluation results.
    
    Args:
        evaluation_result: The result from deepeval.evaluate()
        model: LLM to use for insight generation
    
    Returns:
        InsightResult with summary and recommendations
    """
    # 1. Aggregate metrics across test_results
    metric_stats = _aggregate_metrics(evaluation_result.test_results)
    
    # 2. Generate insight via LLM
    prompt = INSIGHT_PROMPT.format(
        total_testcases=len(evaluation_result.test_results),
        metric_stats=metric_stats,
    )
    
    response = model.generate(prompt)
    return _parse_insight_response(response, metric_stats)
```

### 2. Aggregation logic

Uses only **guaranteed fields** from `MetricData`:
- `name` (str) - metric name
- `threshold` (float) - pass threshold  
- `success` (bool) - whether metric passed

```python
def _aggregate_metrics(test_results: List[TestResult]) -> List[MetricStats]:
    """Aggregate pass rates and failure counts per metric."""
    stats = defaultdict(lambda: {"passes": 0, "total": 0})
    
    for result in test_results:
        for metric in result.metrics_data:
            stats[metric.name]["total"] += 1
            if metric.success:
                stats[metric.name]["passes"] += 1
    
    return [
        MetricStats(
            name=name,
            pass_rate=s["passes"] / s["total"],
            fail_count=s["total"] - s["passes"],
            severity=_determine_severity(s["passes"] / s["total"]),
        )
        for name, s in stats.items()
    ]
```

### 3. Output types

```python
class MetricStats(BaseModel):
    name: str
    pass_rate: float
    fail_count: int
    severity: str  # "critical", "warning", "good"

class Recommendation(BaseModel):
    priority: int
    action: str
    rationale: str
    affected_metrics: List[str]

class InsightResult(BaseModel):
    summary: str
    metric_stats: List[MetricStats]
    recommendations: List[Recommendation]
```

## Optional Enhancement: `description` property on `BaseMetric`

To help the insight LLM understand **what each metric measures**, I suggest adding an optional `description` property:

```python
# In deepeval/metrics/base_metric.py
class BaseMetric:
    # ... existing fields ...
    
    @property
    def __description__(self) -> str:
        """Human-readable description of what this metric measures."""
        return ""  # Default empty, overridden by subclasses
```
  
This would allow the insight generator to produce more contextual recommendations like:
> "Answer Relevancy (measures response relevance to input) failed in 3/10 cases..."

## Design Principles

1. **Non-breaking**: New optional module, doesn't modify existing `evaluate()` behavior
2. **Minimal dependencies**: Only uses guaranteed fields from `EvaluationResult`
3. **Optional**: Users opt-in by calling `generate_insights()` separately
4. **Follows existing patterns**: Uses `DeepEvalBaseLLM` like other deepeval components

## Questions for Maintainers

1. Does this feature align with deepeval's roadmap?
2. Is the `description` property addition to `BaseMetric` acceptable?

Happy to submit a PR if there's interest! Let me know your thoughts.

Hey @rst0070 thanks for your kind words and that deepeval is serving you good - we have this feature on the platform already and I personally don't see us open-sourceing it for now. (we're hiring tho so if you want to work on something like this email hiring@confident-ai.com! )

@penguine-ip Thanks for replying. I totally understand—I didn't realize this overlapped with the platform features. 
I really appreciate the mention regarding hiring. I'll definitely keep it in mind! For now, I'll look for other non conflicting ways to contribute to the repo.