navihat/peer-review-claim-relation-classifier

ReviewSynth NLI — Relation Classifier for Peer Review Claims

Fine-tuned 6-class relation classifier that decides how two atomic claims from different peer reviewers of the same paper relate to each other — e.g. do two reviewers agree, partially agree, flag different symptoms of the same issue, or contradict each other outright.

Base model: MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 (mDeBERTa-v3-base, ≈0.3B params), fine-tuned on 989 hand/LLM-labeled claim pairs.

Full training code, data pipeline, and the FastAPI inference server live at github.com/navihat/peer-review-claim-relations.

---

Label space

AGREEMENT — PARTIAL_AGREEMENT — COMPLEMENTARY — PARTIAL_CONTRADICTION — CONTRADICTION
                                      ⊥
                                  UNRELATED

Label	One-line definition
AGREEMENT	Same specific point, same direction, same strength
PARTIAL_AGREEMENT	Same direction, different scope or certainty
COMPLEMENTARY	Different specific points, but both target the same underlying issue
PARTIAL_CONTRADICTION	Same issue, one hedges where the other is firm
CONTRADICTION	Same specific point, logically incompatible — both cannot be true
UNRELATED	No shared issue

The full decision tree used to label the data is in RUBRIC_GOLD.md in the GitHub repo.

---

How to use

This model classifies a pair of claims, not a single text — that's why the Hub's default inference widget is disabled above (it only supports single-text input, and it can't reproduce the symmetric test-time augmentation this model needs for good accuracy). Use the snippet below instead.

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

LABELS = ["AGREEMENT", "PARTIAL_AGREEMENT", "COMPLEMENTARY",
          "PARTIAL_CONTRADICTION", "CONTRADICTION", "UNRELATED"]

tok   = AutoTokenizer.from_pretrained("navihat/reviewsynth-nli")
model = AutoModelForSequenceClassification.from_pretrained(
            "navihat/reviewsynth-nli").eval()

def predict(left: str, right: str) -> str:
    """Symmetric TTA: average logits over both orderings before argmax."""
    with torch.no_grad():
        logits = sum(
            model(**tok(a, b, return_tensors="pt",
                        truncation=True, max_length=160)).logits
            for a, b in [(left, right), (right, left)]
        ) / 2
    return LABELS[logits.argmax().item()]

print(predict(
    "The method is well-motivated and addresses a real gap.",
    "I fail to see why this approach is needed over prior work."
))
# → CONTRADICTION

Always pass both orderings and average the logits. Skipping symmetric TTA degrades accuracy on order-sensitive examples — order-sensitivity was the root failure mode of the LLM ensemble originally used to build this dataset (flip-rate 44-50% raw vs. ~10% for this fine-tuned model).

A ready-to-run FastAPI server (/predict and a batch /v1/relations:predict endpoint) is in the GitHub repo.

---

Training data

989 claim pairs extracted from ICLR / NeurIPS peer reviews and labeled with Claude using a fixed rubric.

Source	n	How
full_batch_manual	800	Claude reads each pair against rubric
mined_stance_opposition	150	Mined POSITIVE-vs-NEGATIVE pairs, then labeled
fewshot_human_pairs	39	Human-curated examples, labels by Claude
Total	989

Split by paper group to prevent claim leakage between train and eval (train 788 / val 100 / test 101 pairs, 177/15/18 papers respectively).

Label distribution in train:

Label	n	%
UNRELATED	273	34.6
PARTIAL_AGREEMENT	173	22.0
PARTIAL_CONTRADICTION	134	17.0
COMPLEMENTARY	122	15.5
AGREEMENT	61	7.7
CONTRADICTION	25	3.2

The high UNRELATED share reflects how the pairs were mined (same paper, same broad aspect → many claims that share a topic but not a specific issue).

---

Performance

All silver labels are Claude-generated. Numbers measure agreement with those labels, not absolute ground truth.

	macro-F1
Majority baseline (always predict most-common class)	0.10
Stance rule (3-line heuristic)	0.19
Fine-tune — 5-fold CV on silver (989 pairs)	~0.28–0.36
Gold test — 129 pairs, Vietnamese, out-of-domain	0.25

The gold test set contains 129 human-verified pairs from a different domain and language (Vietnamese university grant reviews). The cross-lingual / cross-domain drop from the English test split to this gold set is only ~0.04, which indicates strong multilingual transfer from the mDeBERTa-v3 backbone.

Flip-rate (how often the model changes its prediction when the two claims are swapped) on raw logits: ~10%, down from 44–50% for the raw LLM labelers used during data construction.

Per-class F1 (5-fold CV, silver):

Label	F1
COMPLEMENTARY	0.58
PARTIAL_AGREEMENT	0.48
PARTIAL_CONTRADICTION	0.35
AGREEMENT	0.29
UNRELATED	0.13 †
CONTRADICTION	0.03 †

† UNRELATED F1 is suppressed by prior mismatch (see Limitations below). CONTRADICTION has only 25 train / 2 test samples — read its F1 from CV, not from the test split.

---

Design decisions

Symmetric training + inference. The relation between claim A and claim B must not depend on which is listed first. Three mechanisms enforce this:

Mechanism	Where
Augment both orderings	Training: each pair appears as (A,B) and (B,A) with the same label
Symmetric TTA	Inference: logits from both orderings are averaged before argmax
Flip-rate metric	Measured on raw logits (TTA disabled) to verify the model learned symmetry, not just masked by TTA

Class weights. CrossEntropyLoss(weight=inverse_frequency) so the model does not ignore rare classes. CONTRADICTION receives ~5–6× weight.

Paper-grouped split. Five folds stratified by label, groups defined at the paper level.

---

Limitations

• Labels are LLM-generated. No independent human annotator on the training set → no human agreement ceiling. A macro-F1 of 0.25 cannot be interpreted without knowing inter-annotator agreement on the same task.
• CONTRADICTION is rare (25/989 train samples). F1 on the 2-sample test split is not informative; use the 5-fold CV numbers.
• Prior mismatch on deployment data. UNRELATED is 35% of training data but only 14% of the gold test set. If your deployment distribution differs substantially, consider a prior correction on the logits.
• Training domain: English ML conference reviews. Transfer to other peer-review venues or languages will vary; the gold test (Vietnamese grant reviews, 0.25 macro-F1) is a single data point, not a guarantee.
• No second annotator. The only human annotation is the 129-pair gold set. All 989 training labels are LLM-generated.

---

Citation

@misc{reviewsynth-nli-2026,
  author = {Truong Van Thai},
  title  = {ReviewSynth NLI: 6-class Relation Classifier for Peer Review Claims},
  year   = {2026},
  url    = {https://huggingface.co/navihat/reviewsynth-nli}
}