Repository: confident-ai/deepeval
Issue: Feature: Extend Agent metrics (Task Completion & Tool Correctness) to support multi-turn conversations
Root cause: Unimplemented Feature Gap

**❗BEFORE YOU BEGIN❗**
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

**Is your feature request related to a problem? Please describe.**
Currently, the Task Completion and Tool Correctness metrics are designed only for single-turn conversations using `LLMTestCase`. This limitation prevents users from evaluating agent performance in multi-turn conversational scenarios, which are increasingly common in real-world applications. Users who want to assess how well an agent completes tasks or uses tools across multiple conversation turns have no built-in metrics available, forcing them to either:
1. Break down multi-turn conversations into individual single-turn test cases (losing conversational context)
2. Create custom evaluation logic
3. Use generic conversational metrics that don't specifically focus on task completion or tool usage

**Describe the solution you'd like**
I would like to see conversational versions of the Task Completion and Tool Correctness metrics that can work with `ConversationalTestCase`. Specifically:

**1. Conversational Task Completion Metric:**
- Inherit from `BaseConversationalMetric` instead of `BaseMetric`
- Accept `ConversationalTestCase` without requiring `scenario` or `expected_outcome` fields
- Automatically infer the task goal from the entire conversation context (all turns)
- Evaluate whether the agent successfully completed the task identified from the conversation
- Aggregate tool usage across all turns in the conversation
- Analyze the final conversation outcome based on the complete dialogue history

**Key differences from single-turn version:**
- **Single-turn**: Extracts task and outcome from individual input/output/tools_called
- **Multi-turn**: Infers task intent from complete conversation history and evaluates task completion across the entire dialogue

**2. Conversational Tool Correctness Metric:**
- Inherit from `BaseConversationalMetric` instead of `BaseMetric`
- Accept `ConversationalTestCase` and evaluate tool usage across all turns
- Support evaluation parameters like ordering across the entire conversation
- Compare expected tools (defined at conversation level) with tools called across all turns
- Provide detailed breakdown of tool usage per turn while giving an overall conversation score

**Key features for both metrics:**
- Ability to aggregate tool calls from multiple turns
- Maintain backward compatibility with existing single-turn versions
- Support for both exact matching and fuzzy matching across conversation turns
- Detailed verbose logging showing turn-by-turn analysis
- Flexible design that doesn't require predefined scenarios or expected outcomes

**Describe alternatives you've considered**
1. **Manual aggregation**: Users could manually combine single-turn evaluations, but this loses conversational context and doesn't provide a holistic view of task completion
2. **Custom metrics**: Users could create their own conversational versions, but this duplicates effort and lacks the robustness of the existing implementations
3. **Using generic ConversationalGEval**: While flexible, this doesn't provide the specific task completion and tool correctness logic that these specialized metrics offer

**Additional context**
This enhancement would:
- Complete the agent evaluation toolkit for multi-turn scenarios
- Align with the existing pattern of having both single-turn and conversational versions of metrics
- Support the growing use case of conversational agents that need to complete complex tasks over multiple turns
- Maintain consistency with the existing codebase architecture (BaseConversationalMetric, etc.)

**Implementation considerations:**
- The conversational versions should aggregate `tools_called` from all turns in the conversation
- Task extraction should consider the entire conversation context, not just individual turns
- The metrics should support the existing evaluation parameters and configuration options
- Error handling should account for conversations where some turns may not have tool calls
- The design should be flexible enough to work without requiring predefined scenarios or expected outcomes

**Example use case:**
A customer service agent handling a refund request across multiple turns:
- Turn 1: User requests refund
- Turn 2: Agent asks for order details, calls order lookup tool
- Turn 3: User provides details
- Turn 4: Agent verifies identity, calls verification tool
- Turn 5: Agent processes refund, calls payment tool

The conversational metrics would evaluate:
- **Task Completion**: Whether the refund process was successfully completed based on the conversation flow
- **Tool Correctness**: Whether the right tools were called in the appropriate sequence across all turns


Conversational metrics will be out by today as well!

@danerlt released :)