truthscore-llm 0.1.4
A research library for evaluating truthfulness of LLM outputs based on evidence agreement, self-consistency, and retrieval coverage
Requires Python: >=3.8
Keywords: llm, truth, evaluation, evidence, consistency, research

# TruthScore-LLM

A research-oriented Python library for evaluating the truthfulness of LLM-generated outputs based on evidence agreement, self-consistency, and retrieval coverage.

## Overview

TruthScore-LLM implements a multi-dimensional scoring system that assesses the reliability of answers generated by large language models. The library evaluates answers across four key dimensions:

- **Evidence Agreement**: How well retrieved evidence documents support the answer (entailment checking)
- **Self-Consistency**: Internal coherence and logical consistency of the answer
- **Retrieval Coverage**: Comprehensiveness of supporting evidence
- **Language Confidence**: Linguistic quality and certainty indicators

These component scores are aggregated into a single **truth score** (0.0 to 1.0) and a categorical **decision** (ACCEPT, QUALIFIED, or REFUSE).

## Installation

### PyPI Installation

The library is available on PyPI and can be installed via:

```bash
pip install truthscore-llm
```

### Development Installation

To install the library in development mode:

```bash
git clone https://github.com/mmsa/truthscore-llm.git
cd truthscore-llm
pip install -e .
```

## Quick Start

```python
from truthscore import TruthScorer

# Initialize scorer
scorer = TruthScorer()

# Evaluate an answer
result = scorer.score(
    question="Does vitamin C prevent the common cold?",
    answer="Vitamin C prevents the common cold."
)

# Access results
print(f"Truth Score: {result['truth_score']:.3f}")
print(f"Decision: {result['decision']}")
print(f"Evidence Score: {result['evidence_score']:.3f}")
print(f"Consistency: {result['consistency']:.3f}")
print(f"Language Confidence: {result['language_confidence']:.3f}")
print(f"Coverage: {result['coverage']:.3f}")
```

### Output Format

The `score()` method returns a dictionary with the following structure:

```python
{
    "truth_score": float,          # Overall truth score [0.0, 1.0]
    "decision": str,               # "ACCEPT" | "QUALIFIED" | "REFUSE"
    "evidence_score": float,       # Evidence agreement [0.0, 1.0]
    "consistency": float,          # Self-consistency [0.0, 1.0]
    "language_confidence": float,  # Language confidence [0.0, 1.0]
    "coverage": float              # Retrieval coverage [0.0, 1.0]
}
```

## Configuration

You can customize scoring behavior by providing a custom configuration:

```python
from truthscore import TruthScorer, TruthScoreConfig

# Create custom configuration
config = TruthScoreConfig(
    evidence_weight=0.5,
    consistency_weight=0.3,
    coverage_weight=0.15,
    language_weight=0.05,
    accept_threshold=0.80,
    qualified_threshold=0.60
)

# Initialize scorer with custom config
scorer = TruthScorer(config=config)
```

## Project Structure

```
truthscore/
├── truthscore/           # Main package
│   ├── __init__.py      # Package initialization and exports
│   ├── score.py         # TruthScorer main class
│   ├── config.py        # Configuration management
│   ├── retrieve.py      # Evidence retrieval module
│   ├── nli.py           # Natural Language Inference module
│   ├── consistency.py   # Consistency evaluation module
│   └── coverage.py      # Coverage evaluation module
├── examples/            # Usage examples
│   └── example.py
├── tests/               # Unit tests
│   └── test_score.py
├── README.md            # This file
├── pyproject.toml       # Package metadata
└── setup.cfg            # Setuptools configuration
```

## Running Tests

```bash
python -m pytest tests/
```

Or using unittest:

```bash
python -m unittest tests.test_score
```

## Running Examples

```bash
python examples/example.py
```

## Research Disclaimer

**Important**: This library is provided for research purposes. The scoring mechanisms implemented here are experimental and based on placeholder implementations for key components (retrieval, NLI, consistency checking). 

- The current implementations use simple heuristics and are designed to be **deterministic for testing purposes**.
- **Do not use this library as the sole basis for critical decisions** without:
  - Validating results against domain-specific ground truth
  - Replacing placeholder implementations with production-grade systems
  - Calibrating thresholds for your specific use case
  - Conducting thorough evaluation and error analysis

The library is structured to facilitate easy replacement of placeholder components with real systems (e.g., trained NLI models, vector databases for retrieval, consistency checking systems).

## Contributing

Contributions are welcome! Please ensure that:

- Code follows the existing style and structure
- All tests pass
- New features include appropriate tests
- Documentation is updated

## License

MIT License

## Citation

If you use this library in your research, please cite:

```bibtex
@software{mostafa2025truthscore,
  title={TruthScore-LLM: A Research Library for Evaluating Truthfulness of LLM Outputs},
  author={Mostafa, Mohamed},
  year={2025},
  url={https://github.com/mmsa/truthscore-llm}
}
```