Repository: confident-ai/deepeval
Issue: ConversationRelevancyMetric Removed in DeepEval 3.7.x, Need Official Replacement & Clarification
Root cause: External Dependency Breakage

### Issue Summary
In DeepEval version 3.2.1, `ConversationRelevancyMetric` works correctly for multi-turn conversational evaluation.  
However, in DeepEval 3.7.x and above, this metric is no longer available, and the documentation does not provide any explanation or migration path.

I attempted to switch to `TurnRelevancyMetric`, but it produces **incorrect and inflated scores**, especially for **negative feedback evaluation datasets**.

### What I Expected
- Clear information on whether `ConversationRelevancyMetric` was deprecated or replaced.
- A recommended metric for multi-turn conversation relevancy (task coverage + relevance).
- Similar scoring behavior between multi-turn vs windowed-turn metrics.
- A migration guide or official note in the documentation.

### What Actually Happened
1. ImportError:
```python
ImportError: cannot import name 'ConversationRelevancyMetric' from 'deepeval.metrics'
````

2. `TurnRelevancyMetric` gives **very high scores (0.90+)** for negative samples, even when responses are clearly irrelevant.

3. No documentation explains:

   * why the metric was removed,
   * why TurnRelevancyMetric behaves differently,
   * how to replicate ConversationRelevancyMetric behavior in newer versions.

### Minimal Reproduction

```python
from deepeval.test_case import ConversationalTestCase, Turn
from deepeval.metrics import TurnRelevancyMetric

turns = [
    Turn(role="user", content="What is X?"),
    Turn(role="assistant", content="This is unrelated and incorrect."),
]

test_case = ConversationalTestCase(turns=turns)
metric = TurnRelevancyMetric()
result = metric.measure(test_case)
print(result)
```

This produces a relevance score that is unexpectedly high for irrelevant answers.

### Additional Context

* Positive feedback datasets → Both metrics behave similarly.
* **Negative feedback datasets → TurnRelevancyMetric produces inflated scores**, while ConversationRelevancyMetric (3.2.1) gives more realistic values.
* I need guidance from maintainers on which metric should be used for multi-turn conversation evaluation going forward.

###Key Questions

- Why does TurnRelevancyMetric score negative feedback so highly?
- Is there any proper replacement for ConversationRelevancyMetric in the version 3.7.x and above?
- Do you recommend implementing a custom metric to replicate the older behavior?

Hey @sacjha-star so `ConversationRelevancy` -> `TurnRelevancy` was just a renaming, it shouldn't affect how the metrics are calculated. Here is the exact commit that did the renaming: https://github.com/confident-ai/deepeval/commit/2de21de7c926b2d4b6c3634c93aea350ca1baaab#diff-9a1728e0448f4d0cc2563dcbd989bfa7874141518f001c03c1650e679bc14919

Did you update your evaluation model in any way?

Hi @penguine-ip , 

**Thanks for the reply!**

That’s helpful to know that `ConversationRelevancyMetric` → `TurnRelevancyMetric` was only a rename.

However, I’m still observing a big difference in scores on the *same* dataset, especially for negative feedback conversations.

Here’s what I did:

* **DeepEval 3.2.1**

  * Metric: `ConversationRelevancyMetric`
  * LLM: `gemini-2.5-flash` (same in both runs)
  * Dataset: negative-feedback conversations from our production RAG logs
  * Example scores (average):

    * conversation_relevancy: **~0.65**

* **DeepEval 3.7.x**

  * Metric: `TurnRelevancyMetric`
  * Same testcases, same LLM
  * Example scores (average):

    * turn_relevancy: **~0.96**
  * Even though i tried to add a prompt but still score is around **0.95**

This is on data where the assistant responses are *known bad* (user gave explicit negative ratings), so 0.9+ feels unintuitive.

A few questions to help me debug this:

1. Between 3.2.1 and 3.7.x, did any **defaults** change for Turn/Conversation relevancy? (e.g., `strictness`, or internal evaluation prompt)
2. Is there any **normalization or thresholding** behavior that might have changed between these versions?
3. Is there a recommended **strictness level** or configuration when using TurnRelevancy on negative feedback / failure cases?


Hey @sacjha-star thanks for providing the info, to answer your questions:
1. Not that I know of, I even checked that file using Blame and I don't see any changes
2. Not at all, we still take average with the same configs - the only thing I can think of is maybe we introduced better schema and typing to other models such as Gemini (?) But i dont think it will affect things
3. Not really, it depends what you are comfortable with

A quick test to see if it is the gemini models acting up is to just install the older version and run it with the same configs. 

Hi @penguine-ip,

Thanks for the clarification. I ran the version test as you suggested.

**On a synthetic dataset** scores are

* **3.2.1 → ~0.43**
* **3.7.x → ~0.53**

However, **on my actual evaluation dataset**, the difference is still very large:

* **3.2.1 → ~0.65**
* **3.7.x → ~0.95**

Same LLM, same prompt, same strictness, same testcases.

So it looks like the discrepancy only appears on:

* real multi-turn conversations,
* especially negative-feedback cases where the assistant reply is clearly off-topic.

This makes me wonder if the newer schema/typing or the parsing interface in 3.7.x is causing Gemini to respond more “positively,” producing inflated relevancy scores.

Before I investigate further, do you know if:

1. The evaluation prompt sent to the LLM changed slightly between versions?
2. 3.7.x applies stricter schema enforcement that might influence Gemini’s scoring behavior?
3. Anyone else has seen similar inflation when upgrading?

This will help me narrow down whether the shift is coming from the LLM interaction layer rather than the scoring logic.