merged-mpnet-base

This is a Sentence Transformer model created by a 50/50 linear merge of all-mpnet-base-v2 and multi-qa-mpnet-base-cos-v1, using the SentenceTransformer.merge method from the sentence-transformers library. It maps sentences and paragraphs to 768-dimensional dense vectors for semantic search, clustering, and semantic similarity.

The merged model scores above the average of its two parents on 10 of the 13 NanoBEIR datasets, including 5 where it beats both. It achieves a mean nDCG@10 of 0.5823, compared with 0.5749 for all-mpnet-base-v2 and 0.5712 for multi-qa-mpnet-base-cos-v1, using the same inference settings for all three models.

On the nine datasets evaluated after candidate selection, the merge scores 0.5517, compared with 0.5422 and 0.5384 for its parents.

See Model Creation for the full generation script.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base models: all-mpnet-base-v2 and multi-qa-mpnet-base-cos-v1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Language: English
• License: The all-mpnet-base-v2 model card specifies Apache-2.0. The multi-qa-mpnet-base-cos-v1 repository does not declare a license.

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sentence Transformer Documentation
• Repository: Sentence Transformers on GitHub
• Parent model: all-mpnet-base-v2
• Parent model: multi-qa-mpnet-base-cos-v1

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'MPNetModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/merged-mpnet-base")
queries = ['Which planet is known as the Red Planet?']
documents = [
    "Venus is often called Earth's twin because of its similar size and proximity.",
    'Mars, known for its reddish appearance, is often referred to as the Red Planet.',
    'Saturn, famous for its rings, is sometimes mistaken for the Red Planet.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# (1, 768) (3, 768)

similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.4703, 0.7748, 0.6875]])

Evaluation

Metrics

NanoBEIR

• Dataset: NanoBEIR-en
• Evaluated with NanoBEIREvaluator on all 13 datasets.
• Maximum input length: 512 tokens. Query and document prompts: empty.
• Similarity: cosine. FP32 inference with eager attention.
• Metric: nDCG@10. The mean is unweighted across datasets.

The parent models were evaluated with the same prompts and length limits.

Dataset	merged-mpnet-base	all-mpnet-base-v2	multi-qa-mpnet-base-cos-v1
msmarco	0.609863	0.634465	0.654148
nq	0.665488	0.671772	0.662020
fiqa2018	0.588147	0.583997	0.582674
nfcorpus	0.333769	0.315611	0.313411
scidocs	0.404500	0.454587	0.338733
arguana	0.572705	0.529703	0.557431
dbpedia	0.537960	0.538413	0.555685
quoraretrieval	0.919392	0.921402	0.908255
touche2020	0.483061	0.487039	0.450271
climatefever	0.310998	0.243224	0.333320
fever	0.764325	0.763131	0.745699
hotpotqa	0.638921	0.626516	0.642700
scifact	0.741368	0.703311	0.680758
mean	0.582346	0.574859	0.571162

Model Creation

The complete MPNet backbone was merged in fp32 with equal weights. Mean pooling and L2 normalization are shared by both parents. The maximum input length is 512 tokens, without any additional training.

This is all it took:

# pip install "sentence-transformers[merge]"
from sentence_transformers import SentenceTransformer

models = [
    SentenceTransformer("sentence-transformers/all-mpnet-base-v2"),
    SentenceTransformer("sentence-transformers/multi-qa-mpnet-base-cos-v1"),
]
for model in models:
    model.max_seq_length = 512
model = SentenceTransformer.merge(models, method="linear", parameters={"weight": [0.5, 0.5]}, dtype="float32")
model.similarity_fn_name = "cosine"
model.save_pretrained("merged-mpnet-base")

Framework Versions

• Python: 3.12.3
• Sentence Transformers: 6.2.0
• Transformers: 5.4.0
• PyTorch: 2.11.0+cu128
• Mergekit: 0.1.5
• Datasets: 4.8.5
• Tokenizers: 0.22.2

Citation

BibTeX

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

Mergekit

@inproceedings{goddard-etal-2024-arcees,
    title = "Arcee{'}s {M}erge{K}it: A Toolkit for Merging Large Language Models",
    author = "Goddard, Charles and Siriwardhana, Shamane and Ehghaghi, Malikeh and Meyers, Luke and Karpukhin, Vladimir and Benedict, Brian and McQuade, Mark and Solawetz, Jacob",
    editor = "Dernoncourt, Franck and Preo{\c{t}}iuc-Pietro, Daniel and Shimorina, Anastasia",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2024",
    address = "Miami, Florida, US",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-industry.36",
    doi = "10.18653/v1/2024.emnlp-industry.36",
    pages = "477--485",
}