CrossEncoder based on cross-encoder/ms-marco-MiniLM-L6-v2

This is a Cross Encoder model finetuned from cross-encoder/ms-marco-MiniLM-L6-v2 using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

Model Type: Cross Encoder
Base model: cross-encoder/ms-marco-MiniLM-L6-v2
Maximum Sequence Length: 512 tokens
Number of Output Labels: 1 label
Supported Modality: Text

Model Sources

Documentation: Sentence Transformers Documentation
Documentation: Cross Encoder Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Cross Encoders on Hugging Face

Full Model Architecture

CrossEncoder(
  (0): Transformer({'transformer_task': 'sequence-classification', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'scores', 'architecture': 'BertForSequenceClassification'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("cross_encoder_model_id")
# Get scores for pairs of inputs
pairs = [
    ['skill: manage health care staff', 'skill: manage healthcare staff'],
    ['skill: manage health care staff', 'skill: manage staff'],
    ['skill: manage health care staff', 'skill: manage physiotherapy staff'],
    ['skill: manage health care staff', 'skill: work with nursing staff'],
    ['skill: manage health care staff', 'skill: manage chiropractic staff'],
]
scores = model.predict(pairs)
print(scores)
# [ 8.9768  6.0519  2.0458 -3.0845 -3.2492]

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'skill: manage health care staff',
    [
        'skill: manage healthcare staff',
        'skill: manage staff',
        'skill: manage physiotherapy staff',
        'skill: work with nursing staff',
        'skill: manage chiropractic staff',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Training Details

Training Dataset

Unnamed Dataset

Size: 58,329 training samples
Columns: sentence_A, sentence_B, and label

Approximate statistics based on the first 100 samples:

	sentence_A	sentence_B	label
type	string	string	float
modality	text	text
details	min: 7 tokens mean: 8.14 tokens max: 11 tokens	min: 6 tokens mean: 7.94 tokens max: 15 tokens	min: 0.0 mean: 0.22 max: 1.0

Samples:

sentence_A	sentence_B	label
`skill: manage security systems`	`skill: manage technical security systems`	`1.0`
`skill: manage security systems`	`skill: Physical access management software`	`0.0`
`skill: manage security systems`	`skill: manage theft prevention`	`0.0`

Loss: BinaryCrossEntropyLoss with these parameters:

{
    "activation_fn": "torch.nn.modules.linear.Identity",
    "pos_weight": null
}

Evaluation Dataset

Unnamed Dataset

Size: 7,330 evaluation samples
Columns: sentence_A, sentence_B, and label

Approximate statistics based on the first 100 samples:

	sentence_A	sentence_B	label
type	string	string	float
modality	text	text
details	min: 6 tokens mean: 9.23 tokens max: 14 tokens	min: 6 tokens mean: 8.06 tokens max: 16 tokens	min: 0.0 mean: 0.21 max: 1.0

Samples:

sentence_A	sentence_B	label
`skill: manage health care staff`	`skill: manage healthcare staff`	`1.0`
`skill: manage health care staff`	`skill: manage staff`	`0.0`
`skill: manage health care staff`	`skill: manage physiotherapy staff`	`0.0`

Loss: BinaryCrossEntropyLoss with these parameters:

{
    "activation_fn": "torch.nn.modules.linear.Identity",
    "pos_weight": null
}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 32
num_train_epochs: 2
learning_rate: 2e-05
warmup_steps: 0.1
per_device_eval_batch_size: 64
load_best_model_at_end: True

All Hyperparameters

Click to expand

per_device_train_batch_size: 32
num_train_epochs: 2
max_steps: -1
learning_rate: 2e-05
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_steps: 0.1
optim: adamw_torch_fused
optim_args: None
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 1
average_tokens_across_devices: True
max_grad_norm: 1.0
label_smoothing_factor: 0.0
bf16: False
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: None
trackio_bucket_id: None
trackio_static_space_id: None
per_device_eval_batch_size: 64
prediction_loss_only: True
eval_on_start: False
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: False
hub_private_repo: None
hub_model_id: None
hub_strategy: every_save
hub_always_push: False
hub_revision: None
load_best_model_at_end: True
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 42
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
remove_unused_columns: True
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_static_graph: None
ddp_backend: None
ddp_timeout: 1800
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
warmup_ratio: None
local_rank: -1
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss
0.0110	20	0.4635	-
0.0219	40	0.3676	-
0.0329	60	0.3954	-
0.0439	80	0.3731	-
0.0549	100	0.2560	-
0.0658	120	0.3624	-
0.0768	140	0.2626	-
0.0878	160	0.2477	-
0.0987	180	0.2956	-
0.1097	200	0.1886	-
0.1207	220	0.2467	-
0.1317	240	0.2388	-
0.1426	260	0.2001	-
0.1536	280	0.2283	-
0.1646	300	0.2174	-
0.1755	320	0.1939	-
0.1865	340	0.2160	-
0.1975	360	0.2010	-
0.2084	380	0.1628	-
0.2194	400	0.2163	-
0.2304	420	0.1597	-
0.2414	440	0.1784	-
0.2523	460	0.1708	-
0.2633	480	0.1937	-
0.2743	500	0.1611	-
0.2852	520	0.1324	-
0.2962	540	0.2210	-
0.3072	560	0.1689	-
0.3182	580	0.1659	-
0.3291	600	0.1783	-
0.3401	620	0.1739	-
0.3511	640	0.1571	-
0.3620	660	0.1789	-
0.3730	680	0.1508	-
0.3840	700	0.1454	-
0.3950	720	0.1429	-
0.4059	740	0.1391	-
0.4169	760	0.1523	-
0.4279	780	0.1575	-
0.4388	800	0.1604	-
0.4498	820	0.1453	-
0.4608	840	0.1347	-
0.4717	860	0.1590	-
0.4827	880	0.1463	-
0.4937	900	0.1376	-
0.5047	920	0.1316	-
0.5156	940	0.1598	-
0.5266	960	0.1551	-
0.5376	980	0.1657	-
0.5485	1000	0.1293	-
0.5595	1020	0.1452	-
0.5705	1040	0.1122	-
0.5815	1060	0.0984	-
0.5924	1080	0.1725	-
0.6034	1100	0.1010	-
0.6144	1120	0.1168	-
0.6253	1140	0.1362	-
0.6363	1160	0.1464	-
0.6473	1180	0.1785	-
0.6583	1200	0.1252	-
0.6692	1220	0.1686	-
0.6802	1240	0.1464	-
0.6912	1260	0.1539	-
0.7021	1280	0.0989	-
0.7131	1300	0.1488	-
0.7241	1320	0.1319	-
0.7351	1340	0.1021	-
0.7460	1360	0.1210	-
0.7570	1380	0.1251	-
0.7680	1400	0.1003	-
0.7789	1420	0.1114	-
0.7899	1440	0.1151	-
0.8009	1460	0.1086	-
0.8118	1480	0.1561	-
0.8228	1500	0.1064	-
0.8338	1520	0.1044	-
0.8448	1540	0.1506	-
0.8557	1560	0.0780	-
0.8667	1580	0.1150	-
0.8777	1600	0.1405	-
0.8886	1620	0.0695	-
0.8996	1640	0.1628	-
0.9106	1660	0.1092	-
0.9216	1680	0.1155	-
0.9325	1700	0.0999	-
0.9435	1720	0.1335	-
0.9545	1740	0.1163	-
0.9654	1760	0.1432	-
0.9764	1780	0.0786	-
0.9874	1800	0.1338	-
0.9984	1820	0.1265	-
1.0	1823	-	0.2892
1.0093	1840	0.1222	-
1.0203	1860	0.0966	-
1.0313	1880	0.1005	-
1.0422	1900	0.0586	-
1.0532	1920	0.0774	-
1.0642	1940	0.0859	-
1.0752	1960	0.1290	-
1.0861	1980	0.0840	-
1.0971	2000	0.1117	-
1.1081	2020	0.1329	-
1.1190	2040	0.1140	-
1.1300	2060	0.0865	-
1.1410	2080	0.1245	-
1.1519	2100	0.1399	-
1.1629	2120	0.0980	-
1.1739	2140	0.0879	-
1.1849	2160	0.0877	-
1.1958	2180	0.0837	-
1.2068	2200	0.0906	-
1.2178	2220	0.1072	-
1.2287	2240	0.0856	-
1.2397	2260	0.0687	-
1.2507	2280	0.0945	-
1.2617	2300	0.0759	-
1.2726	2320	0.1195	-
1.2836	2340	0.0753	-
1.2946	2360	0.0901	-
1.3055	2380	0.0973	-
1.3165	2400	0.0701	-
1.3275	2420	0.0905	-
1.3385	2440	0.1302	-
1.3494	2460	0.0892	-
1.3604	2480	0.0969	-
1.3714	2500	0.0915	-
1.3823	2520	0.0683	-
1.3933	2540	0.0863	-
1.4043	2560	0.0962	-
1.4152	2580	0.1067	-
1.4262	2600	0.0646	-
1.4372	2620	0.1181	-
1.4482	2640	0.0971	-
1.4591	2660	0.0678	-
1.4701	2680	0.0802	-
1.4811	2700	0.0707	-
1.4920	2720	0.0859	-
1.5030	2740	0.0675	-
1.5140	2760	0.1365	-
1.5250	2780	0.0630	-
1.5359	2800	0.0715	-
1.5469	2820	0.1017	-
1.5579	2840	0.0642	-
1.5688	2860	0.0461	-
1.5798	2880	0.0928	-
1.5908	2900	0.1716	-
1.6018	2920	0.1067	-
1.6127	2940	0.0888	-
1.6237	2960	0.0713	-
1.6347	2980	0.0958	-
1.6456	3000	0.0633	-
1.6566	3020	0.0832	-
1.6676	3040	0.1103	-
1.6786	3060	0.1050	-
1.6895	3080	0.0691	-
1.7005	3100	0.1026	-
1.7115	3120	0.1004	-
1.7224	3140	0.0760	-
1.7334	3160	0.0735	-
1.7444	3180	0.0852	-
1.7553	3200	0.1119	-
1.7663	3220	0.0899	-
1.7773	3240	0.0775	-
1.7883	3260	0.0509	-
1.7992	3280	0.0924	-
1.8102	3300	0.1293	-
1.8212	3320	0.0698	-
1.8321	3340	0.1157	-
1.8431	3360	0.0757	-
1.8541	3380	0.1373	-
1.8651	3400	0.1009	-
1.8760	3420	0.0689	-
1.8870	3440	0.0940	-
1.8980	3460	0.0988	-
1.9089	3480	0.0727	-
1.9199	3500	0.0864	-
1.9309	3520	0.0942	-
1.9419	3540	0.0597	-
1.9528	3560	0.0955	-
1.9638	3580	0.0833	-
1.9748	3600	0.0991	-
1.9857	3620	0.0669	-
1.9967	3640	0.0916	-
2.0	3646	-	0.3289

The bold row denotes the saved checkpoint.

Training Time

Training: 4.3 minutes
Evaluation: 4.1 seconds
Total: 4.3 minutes

Framework Versions

Python: 3.13.13
Sentence Transformers: 5.5.0
Transformers: 5.8.1
PyTorch: 2.12.0
Accelerate: 1.14.0
Datasets: 5.0.0
Tokenizers: 0.22.2

Additional Resources

Training and Finetuning Reranker Models with Sentence Transformers: the end-to-end guide for training or finetuning Cross Encoder (reranker) models.
Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video reranker models through the same API.
Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: training multimodal Cross Encoders.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}