Repository: explodinggradients/ragas
Issue: UQLM-Ragas integration
Root cause: Unimplemented Feature Gap

**Summary**
Integrate UQLM’s response-level confidence scoring (bounded from 0 to 1) into Ragas via new metrics that calls UQLM `BlackBoxUQ` scorers and/or `WhiteBoxUQ` when token-level logprobs are available. These metrics could support per-scorer confidences and an ensemble confidence, with an optional hallucination risk view as 1 - confidence. This would complement faithfulness and relevancy by providing an effective ground-truth-free signal that is model-agnostic.

**Motivation**

* Many evaluation scenarios lack ground truth or rely on small human-labeled sets. These uncertainty-based metrics help flag risky answers without ground truth.
* UQLM offers a suite of uncertainty scorers that are production-friendly and model-agnostic. Integrating UQLM would give Ragas users a principled hallucination risk score alongside existing metrics.
* Users can correlate uncertainty/confidence with faithfulness to triage failure modes and choose mitigation thresholds.

**About UQLM**

* Repo: [https://github.com/cvs-health/uqlm](https://github.com/cvs-health/uqlm)
* Papers:
  - Approach overview and experiments: [https://arxiv.org/abs/2504.19254](https://arxiv.org/abs/2504.19254) 
  - Software overview: [https://arxiv.org/abs/2507.06196](https://arxiv.org/abs/2507.06196) 
* Highlights: multiple scorer families, simple Python API, supports black-box and white-box signals, ensembles.

**`BlackBoxUQ` overview**
Black-box UQ computes response-level confidence scores by measuring consistency in multiple responses generated from the same prompt. The workflow is displayed below:

<img width="2372" height="629" alt="Image" src="https://github.com/user-attachments/assets/941b1588-e7b2-41b5-a75d-67a81b0c8eef" />

UQLM's black-box workflow can be used in two ways. First, responses can be generated and scored simultaneously:

```python
from uqlm import BlackBoxUQ
bbuq = BlackBoxUQ(llm=llm, scorers=["noncontradiction"])
results = await bbuq.generate_and_score(
    prompts=prompts, # list of prompts
    num_responses=5 # indicates how many sampled responses to generate per prompt
)
```

Second, responses can be scored from pre-generated responses:
```python
bbuq = BlackBoxUQ(scorers=["noncontradiction"])
results = bbuq.score(
    responses=responses, # List of LLM responses
    sampled_responses=sampled_responses,  # List of lists, each with a set of sampled responses to the same prompt
)
```

Available black-box scorers include:
*   Non-Contradiction Probability ([Chen & Mueller, 2023](https://arxiv.org/abs/2308.16175); [Lin et al., 2024](https://arxiv.org/abs/2305.19187); [Manakul et al., 2023](https://arxiv.org/abs/2303.08896))
*   Discrete Semantic Entropy ([Farquhar et al., 2024](https://www.nature.com/articles/s41586-024-07421-0); [Bouchard & Chauhan, 2025](https://arxiv.org/abs/2504.19254))
*   Exact Match ([Cole et al., 2023](https://arxiv.org/abs/2305.14613); [Chen & Mueller, 2023](https://arxiv.org/abs/2308.16175))
*   BERT-score ([Manakul et al., 2023](https://arxiv.org/abs/2303.08896); [Zheng et al., 2020](https://arxiv.org/abs/1904.09675))
*   Cosine Similarity ([Shorinwa et al., 2024](https://arxiv.org/abs/2412.05563); [HuggingFace](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2))


**`WhiteBoxUQ` overview**
* White-box UQ computes response-level confidence scores by measuring consistency in multiple responses generated from the same prompt. The workflow is displayed below:

<img width="1939" height="351" alt="Image" src="https://github.com/user-attachments/assets/6794f8ff-2def-41fb-ad14-8fbadb62fb80" />

White box methods do generation and scoring simultaneously:
```python
from uqlm import WhiteBoxUQ
wbuq = WhiteBoxUQ(llm=llm, scorers=["min_probability"])
results = await wbuq.generate_and_score(prompts=prompts)
```

Available white-box scorers include:

*   Minimum token probability ([Manakul et al., 2023](https://arxiv.org/abs/2303.08896))
*   Length-Normalized Joint Token Probability ([Malinin & Gales, 2021](https://arxiv.org/abs/2002.07650))

**Proposed feature**
Introduce a new set of metrics (or possibly single metric), exposed in `ragas.metrics.experimental` (or similar), which returns:
* uqlm_confidence float in [0, 1], higher means more likely to be correct, *OR*  uqlm_risk float in [0, 1], higher means more likely to contain hallucination (1-confidence). Perhaps this can specified by the user.
* uqlm_high_risk bool when threshold is provided.

**TBD for reviewer discussion**

* Dependency strategy: direct dependency vs optional extra (for example ragas[uqlm]).
* Class design: single class for all scorers vs scorer-specific classes vs separate BlackBox and WhiteBox classes. Preference is to keep BlackBox and WhiteBox separate.
* Naming and default scorer set for the initial release.

**Checklist**
* [ ] Metric classes (polished and well-tested)
* [ ] Dependency (optional?)
* [ ] Documentation page
* [ ] Example notebook
* [ ] Unit and integration tests

**Additional context**
I am the author of UQLM and can help with the API design, implementation, and maintenance on the UQLM side. Happy to open a PR once aligned with Ragas priorities.