Repository: explodinggradients/ragas
Issue: Definition of context precision
Root cause: Documentation Deficiency

Hi, in the official document, I found the context precision is defined as: 
> a metric that evaluates whether all of the **ground-truth** relevant items present in the contexts are ranked higher or not. 

But when calculating the score, its computed by **question**
> This metric is computed using the **question** and the contexts

If the precision is about the gts, shouldn't it be compluted by **gts** and the contexts? In addition, since ragas is evaluating RAG, it is likely that the model itself does not have the relevant knowledge and cannot answer this question. How can we use question to verify context?

Hi @yuukidach , nice observation. As of now, we are checking if the part/chunk of the given text can be used to answer the query or not for context precision. So I suppose we should redefine it that way, What do you think? 

Hi @shahules786 . Thanks for replying. 

I think giving a new calculation formula would be more meaningful than modifying the definition. 

In retrieval task, the precision is usually defined as: 
`precision = (number of relevant docs) / (number of total retrieved docs)`

The key here is how to determine whether a doc is relevant.  If we use questions to confirm whether the context is relevant, we need to assume that the evaluate model can fully answer this question. However, an important scenario in rag is that some private knowledge cannot be fully learned by pre-trained models. This can lead to inaccurate measurements.

I tried to construct a test that included 100 questions: 5 docs were retrieved each time, but only the first two were the ones that truly answered the question. In this case, Ragas gave the context precision of 56 questions a perfect score of 1

Following the current definition of context precision, using gt to measure context precision is actually more intuitive. For example, when we are working on a problem, if we don't know the answer, even if we are given many possible methods, we might still don't know which one is right. But when we have a reference answer, we can filter out effective solutions from a bunch of problem-solving methods

@yuukidach Great, using GT will certainly improve the accuracy of the metric. I want to keep it flexible so that users without access to ground truth also can get some precision-related signals from retrieved chunks. What do you think of a way by which we will automatically switch b/w GT & reference free method depending on the passed dataset for calculating context precision?  

ie, if GT is available -> Use it to calculate CP 
else use the current method. 

@shahules786 The simplest way is to keep the current calculation method and only modify the prompt. I can help with a test to see if modifying the prompt is enough to achieve satisfactory results.

@yuukidach Sure, please give it a go :) 

@shahules786 Hello there,

wanted to ask whats the current behaviour of the metric context precision since the documentation is still talking about questions and contexts but when trying to compute the metric without gts I get the following error:

ValueError: The metric [context_precision] that that is used requires the following additional columns ['ground_truth'] to be present in the dataset. Looks like you're trying to use 'context_precision' without ground_truth. Please use consider using  `context_utilization' instead.

Thx in advance