Repository: truera/trulens
Issue: [INTEGRATION] TruLens Evaluators now available in MLflow GenAI Evaluation
GitHub labels: enhancement

Hi TruLens team!

I've submitted a PR to MLflow that integrates TruLens evaluators as first-class scorers in MLflow's GenAI evaluation framework: https://github.com/mlflow/mlflow/pull/19237

## Complete Usage Example

```python
import mlflow
from mlflow.genai.scorers import (
    TruLensGroundednessScorer,
    TruLensContextRelevanceScorer,
    TruLensAnswerRelevanceScorer,
    TruLensCoherenceScorer,
)

# Define evaluation data
eval_data = [
    {
        "inputs": {"query": "What is the capital of France?"},
        "outputs": "Paris is the capital of France.",
        "context": "France is a country in Western Europe. Its capital city is Paris.",
    }
]

# Run evaluation with TruLens scorers
results = mlflow.genai.evaluate(
    data=eval_data,
    scorers=[
        TruLensGroundednessScorer(),
        TruLensContextRelevanceScorer(),
        TruLensAnswerRelevanceScorer(),
        TruLensCoherenceScorer(),
    ],
)

# View results
print(results.tables["eval_results"])
```

### Example Output

Each scorer returns an MLflow `Feedback` object with consistent structure:

```python
# TruLensGroundednessScorer output
Feedback(
    name="trulens_groundedness",
    value=1.0,
    rationale="reasons: {'criteria': 'Paris is the capital of France.', 
               'supporting_evidence': 'The context states that Paris is the capital city of France.', 
               'score': 1.0}"
)

# TruLensContextRelevanceScorer output  
Feedback(
    name="trulens_context_relevance",
    value=1.0,
    rationale="reason: The context directly answers the query about France's capital 
               by stating 'Its capital city is Paris.'"
)

# TruLensAnswerRelevanceScorer output
Feedback(
    name="trulens_answer_relevance", 
    value=1.0,
    rationale="reason: The response directly and accurately answers the question 
               about the capital of France."
)

# TruLensCoherenceScorer output
Feedback(
    name="trulens_coherence",
    value=1.0,
    rationale="reason: The response is clear, concise, and logically structured."
)
```

## MLflow Scoring Convention Alignment

These TruLens scorers follow **MLflow's standard scoring convention**, ensuring consistency with MLflow's built-in evaluators:

| Score | Meaning |
|-------|---------|
| **1.0** | Best quality (fully grounded, highly relevant, coherent) |
| **0.0** | Worst quality (not grounded, irrelevant, incoherent) |

This matches how MLflow's native scorers like `RetrievalGroundedness`, `RelevanceToQuery`, and `Correctness` behave, allowing users to seamlessly mix TruLens scorers with MLflow's built-in scorers in the same evaluation run.

## Available Scorers

| Scorer | Purpose | Required Fields |
|--------|---------|-----------------|
| `TruLensGroundednessScorer` | Evaluates if outputs are grounded in context | `outputs`, `context` |
| `TruLensContextRelevanceScorer` | Evaluates context relevance to query | `inputs.query`, `context` |
| `TruLensAnswerRelevanceScorer` | Evaluates answer relevance to query | `inputs.query`, `outputs` |
| `TruLensCoherenceScorer` | Evaluates logical flow and structure | `outputs` |

## Features

- **Lazy loading** to avoid import overhead when scorers are not used
- **Configurable model providers** (OpenAI default, LiteLLM supported)
- **Consistent interface** returning MLflow `Feedback` objects
- **Helpful error messages** when `trulens` packages are not installed

## Installation

```bash
# For OpenAI provider (default)
pip install trulens trulens-providers-openai

# For LiteLLM provider (broader model support)
pip install trulens trulens-providers-litellm
```

Would love any feedback from the TruLens team on the integration approach. The PR is currently awaiting review from MLflow maintainers.

Thanks for building such a great evaluation framework!

Hey @debu-sinha - thanks for making this contribution to mlflow!

Could be in a different PR, but would be great to also integrate our goal-plan-action alignment evaluations. These evaluations take the agent trace and identify agent internal errors. We've [benchmarked ](https://arxiv.org/abs/2510.08847)these evaluations against TRAIL with strong results - 95% error coverage compared of TRAIL annotated errors compared to 55% from the LLM judge in the original TRAIL paper.

* [logical consistency](https://github.com/truera/trulens/blob/e901f1cf343078686ddd004b90dcb372d2372a30/src/feedback/trulens/feedback/llm_provider.py#L2533)
* [execution efficiency](https://github.com/truera/trulens/blob/e901f1cf343078686ddd004b90dcb372d2372a30/src/feedback/trulens/feedback/llm_provider.py#L2623)
* [plan adherence](https://github.com/truera/trulens/blob/e901f1cf343078686ddd004b90dcb372d2372a30/src/feedback/trulens/feedback/llm_provider.py#L2713)
* [plan quality](https://github.com/truera/trulens/blob/e901f1cf343078686ddd004b90dcb372d2372a30/src/feedback/trulens/feedback/llm_provider.py#L2801)
* [tool selection](https://github.com/truera/trulens/blob/e901f1cf343078686ddd004b90dcb372d2372a30/src/feedback/trulens/feedback/llm_provider.py#L2889)
* [tool calling](https://github.com/truera/trulens/blob/e901f1cf343078686ddd004b90dcb372d2372a30/src/feedback/trulens/feedback/llm_provider.py#L2975)

AFAICT the required trace data should be available to evaluate as mentioned here https://mlflow.org/docs/3.5.1/genai/eval-monitor/running-evaluation/traces/#step-3-evaluate-trace-quality. Let me know what you think!



Thanks @sfc-gh-jreini - great suggestion! The goal-plan-action alignment evaluations look compelling, especially with the 95% error coverage benchmark against TRAIL.

I'd be happy to add these as a follow-up PR. Looking at the trace data available in MLflow's evaluation API, the structure maps well to what these evaluators need:

**Proposed approach:**
1. Add trace-aware TruLens scorers that consume MLflow's span/trace structure
2. Map MLflow trace spans → TruLens expected format (goal, plan, actions, tool calls)
3. Expose as: `TruLensLogicalConsistencyScorer`, `TruLensExecutionEfficiencyScorer`, `TruLensPlanAdherenceScorer`, `TruLensPlanQualityScorer`, `TruLensToolSelectionScorer`, `TruLensToolCallingScorer`

These would work with `mlflow.genai.evaluate()` when traces are provided, giving users access to your benchmarked agent evaluation capabilities.

A few questions:
1. Do the evaluators expect a specific trace schema, or are they flexible on the input structure?
2. Any minimum trace depth/span requirements for meaningful evaluation?
3. Should I coordinate with anyone on your team for testing/validation?

Will open a separate PR once #19237 lands. Looking forward to collaborating on this.