Repository: confident-ai/deepeval
Issue: Running ConversationalTestCase with ConversationRelevancyMetric is acting strange when evaluating turns
Root cause: Algorithmic Error

### Context 

Current Deepeval version == 3.2.6.
My application is a simple eval tooling for our chat agent. We evaluate entire multi-turn conversations for:

- conversation relevancy
- conversation completeness
- safety (custom conversational GEVal)

When I evaluate any multi turn conversation between user and chatbot as an array of `Turn`, it almost always fails the relevancy score, giving at most 0.5 out of 1.0. And I think I know why.

### To Reproduce

```python
from deepeval.metrics import ConversationRelevancyMetric
from deepeval.evaluate.configs import AsyncConfig
from deepeval.test_case import ConversationalTestCase, Turn
from deepeval.evaluate import evaluate as de_evaluate
chat_role = """
Experienced banking Customer Support Specialist working for a bank in middle east.
Can accurately classify/analyze user inquiries and build relevant responses.
- user is a customer of the bank
- assistant is a conversational chat bot
"""
if __name__ == "__main__":
    conv_case = ConversationalTestCase(
        chatbot_role=chat_role,
        turns=[
            Turn(role="user", content="Hello! I want to open a savings account at your bank."),
            Turn(role="assistant", content="""
                 Hi! You can start the process of opening a new savings account by going to our website and filling out the online form. 
                 You will need to provide the required documents such as your Passport or ID. If you are a non-resident, you will also need to provide address proof and recent bank statements. 
                 If you need further assistance, please let me know.""")
        ]
    )
    de_evaluate(
        test_cases=[conv_case],
        metrics=[
            ConversationRelevancyMetric(
                threshold=0.8, 
                model = 'gpt-4o',
                verbose_mode=True,
            )
        ],
        async_config=AsyncConfig(max_concurrent=1)
    )
```

### Deepeval Logs

```
**************************************************
Conversation Relevancy Verbose Logs
**************************************************

Turns Sliding Windows (size=10):
[
    [Turn(role='user', content='Hello! I want to open a savings account at your bank.', user_id=None, 
retrieval_context=None, tools_called=None, additional_metadata=None)],
    [Turn(role='user', content='Hello! I want to open a savings account at your bank.', user_id=None, 
retrieval_context=None, tools_called=None, additional_metadata=None), Turn(role='assistant', content='\n    
Hi! You can start the process of opening a new savings account by going to our website and filling out the 
online form. \n                 You will need to provide the required documents such as your Passport or ID.
If you are a non-resident, you will also need to provide address proof and recent bank statements. \n       
If you need further assistance, please let me know.', user_id=None, retrieval_context=None, 
tools_called=None, additional_metadata=None)]
] 
 
Verdicts:
[
    {
        "verdict": "no",
        "reason": "There is no assistant message to evaluate for relevancy."
    },
    {
        "verdict": "yes",
        "reason": null
    }
]
 
Score: 0.5
Reason: The score is 0.5 because the absence of an assistant message in message number 1 makes it impossible
to evaluate its relevancy, leading to a neutral score.
```

### EXPECTED BEHAVIOR

I was expecting to get at least 0.9 in Conversation Relevancy score, because there is only 1 turn and assistant response was highly relevant.

**DIAGNOSIS**

But in reality, I get 0.5 consistently. I looked into `ConversationRelevancyMetric` implementation, method  `def measure`. This is the current ConversationalTestCase processing logic: 

- From an array of N Turns,  `ConversationRelevancyMetric` creates N subarrays using method `get_turns_in_sliding_window`. In my case I would get 2 Turn-Sliding Windows with count=1 and count=2.
- Sends each Turn-Sliding Window to Relevancy evaluation (2 LLM-powered eval requests in total)

Of course I would get a score=0.5! 
Because the first Turn-Sliding Window only contains user's question with no assistant response. There's nothing to evaluate in such Turn array. The only exception when sending a single message to evaluation would be if the very first message in conversation is always from chatbot itself.
 
Take a look at my deepeval logs, it specifically gave verdict="no" to it, because it's just a greeting and question from user about opening a savings account.

### Solution options

Option 1: Preprocessing the Turn array in a smarter way - finding ACTUAL turns in them and creating subarrays by isolating turns, so that in each sub array the finishing message would be from assistant. This way the judge LLM will always detect and evaluate the final assistant response.

Option 2: If you want to keep the Turn-Sliding Window generation logic intact, then change the prompt template, so that it ignores the cases where:

- there is not assistant message
- or the last message in subarray is an unanswered question from user. Judge LLM must evaluate the assistant response before that message.

But I really prefer the option A.

Anyway, thank you for building and maintaining Deepeval, you've been a real game changer in our project, guys!
Cheers!  

**Desktop (please complete the following information):**
 - OS: MacOS 14.1 (23B2073)
 - Version Deepeval 3.2.6

@kalmurzayev thanks for the detailed debugging, will fix it next week :)

faced the same issue after migration from 3.0.1 to 3.2.6

Hey @kalmurzayev and @sadomtsev passing this onto @A-Vamshi , option 1 is what we're be making a PR on

Looks like this was fixed? Should we close this issue? Thanks for all the help.

@lputnam-hedgineer this was fixed by PR #1873 . So yeah, my issue can be closed.