EViSE-mmBERT

EViSE-mmBERT is a higher-capacity implementation of the EViSE grouped multi-criteria evaluation framework for Vietnamese text summarization.

The model evaluates a source document–summary pair along three complementary criteria:

• Faithfulness (F): whether the summary is supported by the source.
• Coherence (C): whether the summary is logically organized and linguistically well formed.
• Relevance (R): whether the summary preserves salient source information while avoiding low-value content.

EViSE-mmBERT is used as a secondary framework instantiation to test whether the same grouped cardinal–ordinal learning formulation remains effective under a substantially different encoder, representation operator, and input budget. The primary compact EViSE implementation remains the main model used for the broader OOD and downstream analyses in the accompanying study.

Model overview

Component	Setting
Backbone	jhu-clsp/mmBERT-base
Input	Joint source document–summary pair
Joint input budget	2,048 tokens
Summary cap	512 tokens
Representation	CLS + masked mean pooling
Shared projection	Linear(2H,256) -> GELU -> Dropout(0.1)
Outputs	Faithfulness, Coherence, Relevance
Training objective	Regression + within-document pairwise ranking
Ranking margin	m = 0.05
Ranking-loss weight	lambda = 0.35
Optimizer	AdamW
Learning rate	2e-5
Weight decay	0.01
Warmup ratio	0.05
Maximum epochs	5
Gradient clipping	1.0
Seed	42

Learning formulation

For a source document (D_i), candidate summary (S_i), encoder (E_\theta), and representation operator (P_\psi),

[ H_i^{(L)} = E_\theta(D_i,S_i;L), \qquad z_i = P_\psi(H_i^{(L)}), ]

with criterion-specific predictions

[ \hat y_i^k = f_k(z_i), \qquad k \in {F,C,R}. ]

Training combines cardinal and ordinal supervision:

[ \mathcal{L}_{\mathrm{hybrid}}

\mathcal{L}{\mathrm{reg}} + \lambda\mathcal{L}{\mathrm{rank}}. ]

The regression term calibrates predictions to human-reviewed criterion scores. The ranking term compares candidate summaries only within the same source document, providing relative-quality supervision while avoiding comparisons across sources with different intrinsic difficulty.

The reported implementation uses (m=0.05) and (\lambda=0.35). These are practical operating settings for the reported grouped objective, not universal backbone-independent optima.

Data

The model is trained using the same source-grouped Vietnamese multi-criteria evaluation resource used by EViSE.

The complete resource contains:

• 13,476 source documents
• 6 candidate summaries per source
• 80,856 source–summary evaluations
• human-reviewed Faithfulness, Coherence, and Relevance labels

The split is performed by doc_id to avoid document-level leakage.

Split	Documents	Source–summary pairs
Train	10,915	65,490
Validation	1,213	7,278
Test	1,348	8,088

Dataset:
https://huggingface.co/datasets/phuongntc/EViSE-Dataset

In-domain evaluation

Evaluation is performed on the held-out Vietnamese News test set containing 1,348 source documents and 8,088 source–summary pairs.

Pointwise agreement with human-reviewed scores

Criterion	Spearman rho	Pearson r	MAE
Faithfulness	0.777	0.851	0.096
Coherence	0.712	0.790	0.092
Relevance	0.785	0.867	0.104
Overall	0.806	0.876	0.079

For secondary aggregate analysis,

[ \mathrm{Overall}=0.5F+0.3R+0.2C. ]

No independent holistic Overall label was collected. The Overall score is therefore a derived secondary scalar rather than a fourth directly annotated criterion.

Within-document ordering

Criterion	Kendall tau-b	Pairwise accuracy	Tie-aware Top-1
Faithfulness	0.684	0.908	0.837
Coherence	0.631	0.905	0.950
Relevance	0.694	0.910	0.843
Overall	0.686	0.864	0.727

Criterion specificity

The Pearson cross-criterion matrix remains diagonally dominant:

Prediction	Human F	Human C	Human R
Predicted F	0.851	0.672	0.541
Predicted C	0.677	0.790	0.420
Predicted R	0.557	0.452	0.867

This supports criterion-sensitive prediction, but should not be interpreted as full disentanglement among latent quality dimensions.

Accuracy–efficiency trade-off

EViSE-mmBERT is intentionally a higher-capacity operating point than the compact primary EViSE implementation.

A matched benchmark was conducted on the same 1,344 Vietnamese document–summary pairs using:

• NVIDIA Tesla T4
• FP32 inference
• batch size 4
• 32 warm-up pairs
• three complete runs
• median runtime
• model loading excluded

Model	Parameters	ms/pair	Pairs/s
Primary EViSE	183.952M	55.198	18.117
EViSE-mmBERT	307.334M	94.502	10.582

EViSE-mmBERT achieves stronger in-domain agreement but requires greater computational cost. The comparison should be interpreted as an implementation-level accuracy–efficiency trade-off, not as a controlled backbone-only or context-length-only ablation, because the encoder, representation pooling, and input budget change jointly.

Architecture

Document + Summary
        |
      mmBERT
        |
  +-----+------+
  |            |
 CLS       masked mean
  |            |
  +-----concat-+
        |
 Linear(2H,256)
        |
       GELU
        |
   Dropout(0.1)
        |
  +-----+-----+
  |     |     |
  F     C     R

Inference example

Because the released model uses a custom multi-output evaluator architecture, inference should reconstruct the same architecture used during training.

import torch
import torch.nn as nn
from transformers import AutoModel, AutoTokenizer

BACKBONE = "jhu-clsp/mmBERT-base"
MAX_LEN = 2048
SUMMARY_CAP = 512

tokenizer = AutoTokenizer.from_pretrained(BACKBONE, use_fast=True)


def mean_pool(last_hidden_state, attention_mask):
    mask = attention_mask.unsqueeze(-1).type_as(last_hidden_state)
    summed = (last_hidden_state * mask).sum(dim=1)
    counts = mask.sum(dim=1).clamp(min=1e-9)
    return summed / counts


class EViSEmmBERT(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = AutoModel.from_pretrained(BACKBONE)
        hidden = self.model.config.hidden_size

        self.trunk = nn.Sequential(
            nn.Linear(hidden * 2, 256),
            nn.GELU(),
            nn.Dropout(0.1),
        )

        self.head_faith = nn.Linear(256, 1)
        self.head_coh = nn.Linear(256, 1)
        self.head_rel = nn.Linear(256, 1)

    def forward(self, input_ids, attention_mask, token_type_ids=None):
        model_inputs = {
            "input_ids": input_ids,
            "attention_mask": attention_mask,
        }

        if token_type_ids is not None:
            model_inputs["token_type_ids"] = token_type_ids

        out = self.model(**model_inputs)
        hidden = out.last_hidden_state

        cls_vec = hidden[:, 0]
        mean_vec = mean_pool(hidden, attention_mask)
        z = self.trunk(torch.cat([cls_vec, mean_vec], dim=-1))

        return torch.cat(
            [
                self.head_faith(z),
                self.head_coh(z),
                self.head_rel(z),
            ],
            dim=1,
        )

For exact reproduction, use the tokenizer/checkpoint files and train_config.json distributed with this repository. The source document and summary should be paired using the same dynamic token-budget policy used during training: the summary is capped first, then the remaining joint budget is allocated to the source document.

Intended use

EViSE-mmBERT is intended for:

• Vietnamese summarization evaluation
• criterion-level analysis of Faithfulness, Coherence, and Relevance
• evaluator benchmarking and model selection
• research on grouped cardinal–ordinal supervision
• research on accuracy–efficiency trade-offs across evaluator implementations

The model is reference-free at inference time: it requires the source document and candidate summary, not a gold/reference summary.

Limitations

• The reported evaluation is primarily Vietnamese News in-domain evaluation.
• The mmBERT backbone is multilingual, but this checkpoint has not been shown to provide multilingual evaluation generalization.
• OOD and downstream-reward results reported for the primary compact EViSE checkpoint should not be automatically transferred to EViSE-mmBERT.
• The secondary implementation jointly changes encoder, pooling, capacity, and context budget, so performance differences cannot be causally attributed to any single factor.
• Human quality dimensions are correlated. Diagonal dominance of the criterion matrix indicates criterion sensitivity rather than full statistical independence.
• The model should not be treated as a substitute for expert human assessment in high-stakes domains.

Relationship to EViSE

EViSE-mmBERT belongs to the same methodological family as the primary EViSE model.

The framework-level invariants are:

1. source-conditioned shared representation,
2. separate Faithfulness, Coherence, and Relevance prediction,
3. source-grouped training,
4. cardinal regression,
5. within-source ordinal ranking,
6. single-pass multi-criteria inference.

The encoder family, pooling operator, tokenizer, input budget, and computational cost are implementation choices.

Primary compact EViSE model:
https://huggingface.co/phuongntc/Multi_EvalSumViet2

Citation

If you use this model, please cite the accompanying EViSE study when its bibliographic record becomes available.

Until then, the model repository can be cited as:

@misc{Tran2026EViSEmmBERT,
  title        = {EViSE-mmBERT: A Higher-Capacity Instantiation of the EViSE Multi-Criteria Summarization Evaluation Framework},
  author       = {Tran, Thi Thu Phuong and Vu, Trinh Hoang and Van, Vinh Nguyen and Phuong, Thai Nguyen and Nguyen-Duc, Anh-Quan},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  url          = {https://huggingface.co/quancute/mmevalsumviet2-mmbert-2048}
}

Acknowledgements

EViSE-mmBERT builds on the pretrained mmBERT-base encoder from JHU CLSP and the EViSE Vietnamese multi-criteria summarization evaluation framework.

Base model:
https://huggingface.co/jhu-clsp/mmBERT-base