davidvicente/ModernBERT-paper-classifier

ModernBERT Paper Classifier for Deduplication

This model is a fine-tuned version of answerdotai/ModernBERT-base designed to identify duplicate academic papers. It uses a SentenceTransformer architecture with a custom linear classification head trained using SoftmaxLoss.

It was specifically developed to deduplicate records between PubMed and Crossref databases.

Model Details

• Model Type: Sentence Transformer with Binary Classifier
• Base Model: answerdotai/ModernBERT-base
• Tokenizer: ModernBERT Tokenizer
• Classifier: A linear layer (saved as classifier_weights.pt) that takes the concatenation of $(u, v, |u - v|)$ embeddings.
• Maximum Sequence Length: 8192 tokens (inherited from ModernBERT)
• Training Language: English

Intended Use

This model is intended for academic paper deduplication. It expects a specific input format where paper metadata is concatenated using [SEP] tokens.

Input Format

The model was trained on strings joined in the following order: Year [SEP] First Author [SEP] Journal [SEP] Title [SEP] Abstract

Training Data

The dataset was generated by cross-referencing papers from PubMed and Crossref:

1. Positive Pairs: Papers identified as the same via DOI/PMID matches across both databases.
2. Negative Pairs: Challenging negatives were generated by querying the Elasticsearch index of one source (e.g., PubMed) using the title of a paper from the other source (e.g., Crossref).
3. Mirroring: The process was performed in both directions (PubMed → Crossref and Crossref → PubMed).
4. Balance: The final test set was balanced to reflect an estimated real-world scenario where duplicates and non-duplicates appear with similar frequency in high-recall candidate sets.

Training Procedure

• Framework: sentence-transformers
• Loss Function: SoftmaxLoss with 2 output labels (0: Not Duplicate, 1: Duplicate).
• Epochs: 10
• Batch Size: 8
• Learning Rate: 5e-6
• Warmup Steps: 1000
• Optimizer: AdamW

Evaluation Results

Evaluated on a balanced test set:

Metric	Value
Accuracy	0.9862
F1 Score	0.9863
Precision	0.9822
Recall	0.9905

Usage

Since this model includes a separate classifier head, it is best loaded using the snapshot_download from huggingface_hub alongside the PaperClassifier class provided in the accompanying tool.

from huggingface_hub import snapshot_download
from paper_deduplication.paper_classifier import PaperClassifier

# 1. Download model and classifier weights
repo_id = "your-username/ModernBERT-paper-classifier"
model_path = snapshot_download(repo_id)

# 2. Initialize
classifier = PaperClassifier(model_path)

# 3. Predict
is_duplicate = classifier.deduplicate(
    "2023 [SEP] Smith J [SEP] Nature [SEP] Title of paper A [SEP] Abstract...",
    "2023 [SEP] Smith J [SEP] Nature [SEP] Title of paper B [SEP] Abstract..."
)

Citation

If you use this model in your research, please cite the repository:

@software{modernbert_paper_dedup,
  author = {David Vicente},
  institution = {University of Geneva},
  title = {ModernBERT Paper Classifier for Deduplication},
  year = {2026},
  url = {https://huggingface.co/your-username/ModernBERT-paper-classifier}
}