Repository: confident-ai/deepeval
Issue: ValueError: Number of verdicts generated does not equal length of retrieved nodes.
Root cause: Validation Gap

Greetings,

I'm currently experimenting with the example API to analyze my dataset.

```python
evaluator = RAGASEvaluator(
    actual_output=....,
    retrieval_context=....,
    expected_output=....,
    input_question=....,
)
```

During this process, I encountered an issue: a `ValueError` indicating that the count of generated verdicts doesn't match the number of nodes retrieved.


```  File "/Users/jiajingc/anaconda3/lib/python3.11/site-packages/deepeval/metrics/contextual_precision.py", line 48, in measure
    self._generate_verdicts(
  File "/Users/jiajingc/anaconda3/lib/python3.11/site-packages/deepeval/metrics/contextual_precision.py", line 136, in _generate_verdicts
    raise ValueError(
ValueError: Number of verdicts generated does not equal length of retrieved nodes.
```
When I shorten the `retrieval_context`, this problem doesn't occur. However, I aim to utilize the full, extended `retrieval_context`. My questions are as follows:
1. What causes the `ValueError` when using a lengthy `retrieval_context`? What does this error signify?
2. How can I maintain the original `retrieval_context` size while ensuring the evaluator functions correctly?

@Jiajing-Chen So this is happening because the context is so large such that your LLM is not able to generate a verdict for each retrieval node. For some of the RAG metrics, we ask the LLM to generate verdicts for each node and if the context gets too big the LLM might not carry out the task correctly. 

Which LLM are you using? Using a different model usually helps.

Also, I don't recognize the RAGAS evaluator, and it looks extremely neat! It looks like one of our LlamaIndex evaluators, did you create it yourself? 

Hi Jeffrey @penguine-ip, thanks for your reply.  Thanks for letting me know that the reason for the error is that the context is too large. I am using "gpt-4-1106-preview". Do you have any suggestions about which LLM model is able to support longer context? 
Also the RAGAS evaluator is just a very simple wrapper based on your DeepEval I created, thanks for asking!


I see @Jiajing-Chen, so the thing with these contextual metrics is we need to strictly determine whether **each** node is relevant or not. We can't just skip a node or "forget" to consider a node otherwise the calculation for precision and recall wouldn't work, which is why unlike faithfulness and answer relevancy we strictly check for matching verdict and retrieval context lengths. The model seems fine, it might just be theres a lot of context for gpt-4 to digest.

What I could do to make it better is to first compress/pre-process each node, then use the compressed version to see whether or not each node is relevant to the task at hand. This would shorten the context and make it easier for gpt-4 to digest. Do you have a test case to share? It would help me debug in development. 

Also we're on discord if you have it: https://discord.com/invite/a3K9c8GRGt

Hi @penguine-ip , thanks for your reply with detailed explanation! I am more than happy to join the discord channel and share the test case, but the link you shared seems to be invalid. XD

@Jiajing-Chen  wait really? try to click it again... 

https://discord.com/invite/a3K9c8GRGt

It looks the same and it works for me, don't worry about it (although you can also find the link on our GitHub readme)