Repository: explodinggradients/ragas
Issue: [Feature Request] Calculate confidence intervals on metrics
Root cause: Unimplemented Feature Gap

Hi, I'd like to request a feature to calculate confidence intervals on metrics based on the number of samples that are provided. I'm curious if you think this is suitable?

```python
import numpy as np
from scipy import stats
from scipy.stats import shapiro, probplot
import matplotlib.pyplot as plt


def evaluate_recall_precision_f1(examples):
    """
    Evaluate recall, precision, and F1 score for each example in examples using batch processing.
    """
    recalls = []
    precisions = []
    f1_scores = []

    print(f"Getting predictions for {len(examples)} examples...")
    predictions = qa_chain.batch(examples)

    print("Calculating recall, precision, and F1 scores...")
    recall_results = context_recall_chain.evaluate(examples, predictions)
    precision_results = context_rel_chain.evaluate(examples, predictions)

    for recall_result, precision_result in zip(recall_results, precision_results):
        recall = recall_result["context_recall_score"]
        precision = precision_result["context_precision_score"]
        f1 = 2 * (recall * precision) / (recall + precision) if (recall + precision) != 0 else 0

        recalls.append(recall)
        precisions.append(precision)
        f1_scores.append(f1)

    return recalls, precisions, f1_scores



def check_normality(data, alpha=0.05):
    """
    Check if the data follows a normal distribution.

    Parameters:
    - data: array-like, data to test
    - alpha: significance level for Shapiro-Wilk test

    Returns:
    - True if data seems to follow a normal distribution, False otherwise
    """

    # Shapiro-Wilk test
    stat, p = shapiro(data)
    if p > alpha:
        normality = True
    else:
        normality = False

    # Histogram
    plt.figure()
    plt.hist(data, bins=30)
    plt.title("Histogram")
    plt.show()

    # Q-Q plot
    plt.figure()
    probplot(data, plot=plt)
    plt.title("Q-Q Plot")
    plt.show()

    return normality


def calculate_confidence_interval(data, confidence=0.95):
    """
    Calculate the confidence interval for a given set of data.

    The confidence interval provides a range within which the true population parameter is likely to fall with 
    a certain level of confidence. The choice of Z-distribution or T-distribution is based on the sample size.
    For large samples (n >= 30), the sampling distribution of the sample mean is approximately normal, 
    hence the Z-distribution is used. For smaller samples (n < 30), and when the population standard 
    deviation is unknown, the T-distribution is used.

    Parameters:
    - data: list or numpy array of data points.
    - confidence: Desired confidence level. Default is 0.95 for a 95% confidence interval.

    Returns:
    - (mean, standard deviation, lower bound, upper bound): tuple containing the sample mean, 
    sample standard deviation, and the lower and upper bounds of the confidence interval.

    Rationale for Z vs. T distribution:
    - Z-distribution (normal distribution) is used when the sample size is large (usually n ≥ 30), 
    due to the central limit theorem ensuring that the sampling distribution of the mean is 
    approximately normal.
    - T-distribution is used for smaller samples (n < 30) when the population standard deviation 
    is unknown. This distribution accounts for increased variability in smaller samples.

    When considering metrics like recall and precision, especially from natural language QA pairs, there's a 
    question of whether these metrics follow a normal distribution. In many cases, they may not inherently follow 
    a normal distribution, given that recall and precision values are bound between 0 and 1 and can exhibit skewed distributions. 
    However, with a sufficiently large sample size, the central limit theorem suggests that the distribution of the sample mean 
    (e.g., average recall or precision across multiple QA pairs) tends toward a normal distribution. 
    This allows us to leverage the efficiency and direct estimates provided by parametric methods.

    Furthermore, using parametric methods, such as Z or T distributions, for computing confidence intervals is straightforward 
    and computationally less intensive compared to non-parametric methods like bootstrapping. This becomes particularly important 
    when working with large datasets or when performing multiple evaluations in real-time scenarios.

    That said, it's crucial to conduct preliminary data analysis. If the distribution of recall and precision values is found 
    to be heavily skewed or exhibiting extreme outliers, even with a large sample size, it might warrant the consideration of 
    non-parametric methods. On the other hand, if the distribution appears to be reasonably symmetrical or if the 
    central limit theorem can be invoked due to a large enough sample, then parametric methods are justifiable and advantageous 
    in terms of efficiency and simplicity.

    Steps:
    1. Calculate the sample mean and sample standard deviation.
    2. Determine the number of data points in the sample.
    3. Compute the Standard Error of the Mean (SEM).
    4. Decide whether to use the Z-distribution or the T-distribution based on the sample size.
    5. Calculate the margin of error using the chosen distribution.
    6. Determine the lower and upper bounds of the confidence interval using the sample mean and 
    margin of error.
    """

    is_normal = check_normality(data)
    if not is_normal:
        print("Warning: Data does not seem to follow a normal distribution. Consider non-parametric methods.")

    # Step 1: Compute the sample mean and standard deviation
    mean = np.mean(data)
    std_dev = np.std(data, ddof=1)  # Use ddof=1 for sample standard deviation

    # Step 2: Determine the number of data points
    n = len(data)

    # Step 3: Compute the Standard Error of the Mean (SEM)
    sem = std_dev / np.sqrt(n)

    # Step 4: Choose the appropriate distribution based on sample size
    if n >= 30:
        # Use the Z-distribution for larger samples
        z_value = stats.norm.ppf(1 - (1 - confidence) / 2)  # Z value from normal distribution
        # Step 5: Calculate the margin of error using Z-distribution
        margin_error = z_value * sem
    else:
        # Use the T-distribution for smaller samples
        t_value = stats.t.ppf(1 - (1 - confidence) / 2, df=n-1)  # T value for given degrees of freedom
        # Step 5: Calculate the margin of error using T-distribution
        margin_error = t_value * sem

    # Step 6: Determine the bounds of the confidence interval
    lower_bound = mean - margin_error
    upper_bound = mean + margin_error

    return mean, std_dev, lower_bound, upper_bound


# Sample evaluations
examples = [
    {"query": q, "ground_truths": [eval_answers[i]]}
    for i, q in enumerate(eval_questions)
]

recalls, precisions, f1_scores = evaluate_recall_precision_f1(examples)


# Calculate confidence intervals for recall, precision, and F1
confidence = 0.95
recall_mean, recall_std_dev, recall_lower, recall_upper = calculate_confidence_interval(recalls, confidence)
precision_mean, precision_std_dev, precision_lower, precision_upper = calculate_confidence_interval(precisions, confidence)
f1_mean, f1_std_dev, f1_lower, f1_upper = calculate_confidence_interval(f1_scores, confidence)

print(f"Recall mean: {recall_mean:.3f}, Standard Deviation: {recall_std_dev:.3f}, Confidence interval: ({recall_lower:.3f}, {recall_upper:.3f})")
print(f"Precision mean: {precision_mean:.3f}, Standard Deviation: {precision_std_dev:.3f}, Confidence interval: ({precision_lower:.3f}, {precision_upper:.3f})")
print(f"F1 mean: {f1_mean:.3f}, Standard Deviation: {f1_std_dev:.3f}, Confidence interval: ({f1_lower:.3f}, {f1_upper:.3f})")
```

Hi @austinmw , thanks for raising the issue and putting a lot of thought into this. I'm sure that analysis based on confidence interval can be very useful to get an idea of the range of values one can expect if one redo the experiment( scoring here). But this can be calculated using a single function using scipy.

```
import numpy as np 
import scipy.stats as st 

# calculate any ragas score 
ragas_score = evaluate(dataset)['ragas_score']

# create 99% confidence interval 
st.t.interval(alpha=0.99, 
		df=len(ragas_score)-1, 
		loc=np.mean(ragas_score), 
		scale=st.sem(ragas_score)) 
```
Since the same can be achieved easily using Scipy, I'm not sure if we should reimplement that into ragas. 

Makes sense, thanks for taking a look!