Hugging Face's logo

multi-vector-encoder-testing
/
bert-tiny-multi-vector

Feature Extraction
sentence-transformers
Safetensors
English
bert
multi-vector
colbert
late-interaction
Generated from Trainer
dataset_size:501907
loss:MultiVectorMultipleNegativesRankingLoss
Eval Results (legacy)
text-embeddings-inference
Model card Files
xet
 Community
How to use from the
Use from the
sentence-transformers library
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("multi-vector-encoder-testing/bert-tiny-multi-vector")

sentences = [
    "The weather is lovely today.",
    "It's so sunny outside!",
    "He drove to the stadium."
]
embeddings = model.encode(sentences)

similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

BERT tiny multi-vector encoder trained on MS MARCO

This is a Multi-Vector Encoder model finetuned in two stages from prajjwal1/bert-tiny on the msmarco-bm25 dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

Model Details

Model Description

  • Model Type: Multi-Vector Encoder
  • Base model: prajjwal1/bert-tiny
  • Maximum Sequence Length: 512 tokens
    • Maximum Query Length: 32 tokens
    • Maximum Document Length: 256 tokens
  • Output Dimensionality: 128 dimensions
  • Similarity Function: MaxSim
  • Supported Modality: Text
  • Training Dataset:
  • Language: en
  • License: mit

Model Sources

Full Model Architecture 

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_length': 32, 'document_length': 256, 'query_expansion': {'strategy': 'min', 'attend': False, 'token': None, 'length': 32}, 'architecture': 'BertModel'})
  (1): Dense({'in_features': 128, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
  (2): MultiVectorMask({'skiplist_words': [], 'skiplist_tasks': ['document'], 'keep_only_token_ids': None})
  (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("multi-vector-encoder-testing/bert-tiny-multi-vector")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
    'calories in kirkland ravioli',
]
documents = [
    'There are 140 calories in a 4 pieces serving of Kirkland Signature Four Cheese Ravioli. Calorie breakdown: 45% fat, 31% carbs, 24% protein.',
    'There are 140 calories in a 4 pieces serving of Kirkland Signature Four Cheese Ravioli. Calorie breakdown: 45% fat, 31% carbs, 24% protein.',
    'Current Local Time: Cleveland, Ohio is in the Eastern Time Zone: The Current Time in Cleveland, Ohio is: Thursday 1/18/2018 10:41 PM EST Cleveland, Ohio is in the Eastern Time Zone',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (32, 128) (39, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[25.4110, 25.4110,  8.3390]])

Evaluation

Metrics

Multi Vector Information Retrieval

  • Datasets: NanoMSMARCO, NanoNQ, NanoFiQA2018, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
  • Evaluated with MultiVectorInformationRetrievalEvaluator
Metric NanoMSMARCO NanoNQ NanoFiQA2018 NanoClimateFEVER NanoDBPedia NanoFEVER NanoHotpotQA NanoNFCorpus NanoQuoraRetrieval NanoSCIDOCS NanoArguAna NanoSciFact NanoTouche2020
maxsim_accuracy@1 0.16 0.28 0.26 0.18 0.62 0.56 0.72 0.42 0.74 0.26 0.18 0.48 0.3878
maxsim_accuracy@3 0.32 0.4 0.4 0.3 0.78 0.72 0.9 0.52 0.88 0.42 0.38 0.58 0.7755
maxsim_accuracy@5 0.42 0.48 0.48 0.34 0.84 0.82 0.92 0.52 0.9 0.52 0.42 0.6 0.8776
maxsim_accuracy@10 0.7 0.66 0.58 0.5 0.94 0.86 0.96 0.6 0.92 0.74 0.56 0.68 0.9796
maxsim_precision@1 0.16 0.28 0.26 0.18 0.62 0.56 0.72 0.42 0.74 0.26 0.18 0.48 0.3878
maxsim_precision@3 0.1067 0.1333 0.18 0.1 0.4733 0.2467 0.4133 0.3267 0.36 0.18 0.1267 0.2133 0.449
maxsim_precision@5 0.084 0.096 0.132 0.076 0.448 0.172 0.268 0.268 0.228 0.152 0.084 0.136 0.4408
maxsim_precision@10 0.07 0.066 0.08 0.058 0.392 0.092 0.148 0.224 0.12 0.118 0.056 0.078 0.3551
maxsim_recall@1 0.16 0.27 0.1229 0.0917 0.0521 0.5267 0.36 0.0447 0.654 0.054 0.18 0.445 0.0245
maxsim_recall@3 0.32 0.39 0.251 0.13 0.117 0.6767 0.62 0.0718 0.8587 0.11 0.38 0.565 0.096
maxsim_recall@5 0.42 0.46 0.3117 0.16 0.1668 0.7833 0.67 0.0835 0.886 0.154 0.42 0.59 0.1525
maxsim_recall@10 0.7 0.61 0.387 0.2383 0.2876 0.8233 0.74 0.1061 0.9127 0.24 0.56 0.67 0.2408
maxsim_ndcg@10 0.3859 0.43 0.3027 0.1919 0.482 0.6793 0.684 0.2852 0.8355 0.2217 0.3539 0.565 0.392
maxsim_mrr@10 0.292 0.3833 0.3604 0.2663 0.7217 0.6486 0.807 0.4765 0.8162 0.3871 0.2902 0.5397 0.5977
maxsim_map@100 0.3049 0.3841 0.2447 0.1495 0.3543 0.6349 0.602 0.1259 0.8099 0.1561 0.3044 0.5384 0.2962

Multi Vector Nano BEIR

  • Dataset: NanoBEIR_mean
  • Evaluated with MultiVectorNanoBEIREvaluator with these parameters:
    {
    "dataset_names": [
        "msmarco",
        "nq",
        "fiqa2018"
    ],
    "dataset_id": "sentence-transformers/NanoBEIR-en"
}
Metric Value
maxsim_accuracy@1 0.2333
maxsim_accuracy@3 0.3733
maxsim_accuracy@5 0.46
maxsim_accuracy@10 0.6467
maxsim_precision@1 0.2333
maxsim_precision@3 0.14
maxsim_precision@5 0.104
maxsim_precision@10 0.072
maxsim_recall@1 0.1843
maxsim_recall@3 0.3203
maxsim_recall@5 0.3972
maxsim_recall@10 0.5657
maxsim_ndcg@10 0.3729
maxsim_mrr@10 0.3452
maxsim_map@100 0.3112

Multi Vector Nano BEIR

  • Dataset: NanoBEIR_mean
  • Evaluated with MultiVectorNanoBEIREvaluator with these parameters:
    {
    "dataset_names": [
        "climatefever",
        "dbpedia",
        "fever",
        "fiqa2018",
        "hotpotqa",
        "msmarco",
        "nfcorpus",
        "nq",
        "quoraretrieval",
        "scidocs",
        "arguana",
        "scifact",
        "touche2020"
    ],
    "dataset_id": "sentence-transformers/NanoBEIR-en"
}
Metric Value
maxsim_accuracy@1 0.4037
maxsim_accuracy@3 0.5673
maxsim_accuracy@5 0.626
maxsim_accuracy@10 0.7446
maxsim_precision@1 0.4037
maxsim_precision@3 0.2545
maxsim_precision@5 0.1988
maxsim_precision@10 0.1429
maxsim_recall@1 0.2297
maxsim_recall@3 0.3528
maxsim_recall@5 0.4044
maxsim_recall@10 0.5012
maxsim_ndcg@10 0.4468
maxsim_mrr@10 0.5067
maxsim_map@100 0.3773

Training Details

Training Dataset

msmarco-bm25

  • Dataset: msmarco-bm25 at ce8a493
  • Size: 501,907 training samples
  • Columns: query, positive, and negative
  • Approximate statistics based on the first 100 samples:
    query positive negative
    type string string string
    modality text text text
    details
    • min: 4 tokens
    • mean: 8.81 tokens
    • max: 16 tokens
    • min: 25 tokens
    • mean: 78.08 tokens
    • max: 174 tokens
    • min: 22 tokens
    • mean: 76.59 tokens
    • max: 180 tokens
  • Samples:
    query positive negative
    sociopath define Updated September 08, 2016. Both psychopaths and sociopaths are defined as someone who is suffering from Antisocial Personality Disorder. Both groups show a pervasive pattern of disregard for the rights and feelings of others. There are, however, subtle differences between the two groups. Define sociopath. sociopath synonyms, sociopath pronunciation, sociopath translation, English dictionary definition of sociopath. n. A psychopath or a person with antisocial personality disorder. so′ci·o·path′ic adj. so′ci·op′a·thy n. n psychiatry another name for psychopath...
    what county is tarrytown ny in ABOUT US. The Music Hall, an 1885 landmark in Tarrytown, NY is Westchester County's oldest theater and one of the region's busiest music venues, welcoming 85,000 visitors every year, including tens of thousands of children. Please complete all required fields! Tarrytown, NY Other Information. 1 Located in WESTCHESTER County, New York. 2 Tarrytown, NY is also known as: 3 N TARRYTOWN, NY. NORTH TARRYTOWN, 1 NY. PHILIPSE MANOR, 2 NY. POCANTICO HILLS, 3 NY. SLEEPY HOLLOW, NY. SLEEPY HOLLOW MANOR, NY.
    what temperature do you grill a t-bone at Step 2. Move your T-bones to the medium heat side of your grill and continue grilling. If you like your T-bone medium rare, grill for four to five minutes on each side or until a meat thermometer reads 130 to 140 degrees Fahrenheit.For a medium steak, grill six to seven minutes per side or until a meat thermometer reads 140 to 150 degrees.iming and technique are the keys to grilling a 1-inch-thick T-bone steak to perfection. This thicker cut requires different treatment from a T-bone, for example, 1/2- to 3/4-inch thick. Preheat your grill using two temperature settings. If you are using a gas grill, set one side to high and the other to a medium setting, then close the lid for 10 to 15 minutes.iming and technique are the keys to grilling a 1-inch-thick T-bone steak to perfection. This thicker cut requires different treatment from a T-bone, for example, 1/2- to 3/4-inch thick.
  • Loss: MultiVectorMultipleNegativesRankingLoss with these parameters:
    {
    "scale": 1.0,
    "similarity_fct": "colbert_scores",
    "mini_batch_size": null,
    "score_mini_batch_size": null,
    "gather_across_devices": false
}

Evaluation Dataset

msmarco-bm25

  • Dataset: msmarco-bm25 at ce8a493
  • Size: 1,024 evaluation samples
  • Columns: query, positive, and negative
  • Approximate statistics based on the first 100 samples:
    query positive negative
    type string string string
    modality text text text
    details
    • min: 4 tokens
    • mean: 8.65 tokens
    • max: 15 tokens
    • min: 21 tokens
    • mean: 80.76 tokens
    • max: 174 tokens
    • min: 16 tokens
    • mean: 79.88 tokens
    • max: 224 tokens
  • Samples:
    query positive negative
    what is the bottom lip piercing called 1 Standard Lip Piercing – This is a single piercing done off centered on the lower lip. 2 A captive bead ring (CBR) or labret stud can be worn. 3 Monroe Piercing – This is a single piercing done on the left side of the upper lip and is named for the mole on Marilyn Monroe’s lip. 4 Usually a labret stud is worn in this piercing. Also known as lower-lip piercing or bottom lip piercing .The labret piercing is placed at the labrum (below bottom lip, above chin). Popular among men and women, this style looks super cool and trendy.
    what are the mind and body This is known as dualism. Dualism is the view that the mind and body both exist. There are two basic types of dualism: o Descartes dualism: The view that the mind and body function separately, without interchange. o Cartesian dualism argues that there is a two-way interaction between mental and physical substances. Dualism is in contrast to monism that states the mind and body are the same thing. Quotes About What Matters In Life. “What is in your mind position or disposition your mind, body and spirit in the best or worst way. What you are yet to accept into your mind exposes your mind to and keep your mind on what you are yet to accept and what has not yet come into your mind least controls your mind, body and spirit. Browse By Tag.
    what breed of dogs have green eyes Best Answer: There are many dog breeds that CAN have green eyes, such as Australian Shepherds, Border Collies, Siberian Huskies, and others, but it is an uncommon occurrence. However it won't be a bright green like a cat's eye. It'll be a somewhat subdued shade of blueish-grey with green overtones. What are some dog breeds that have or can have green eyes? What breed is this dog? What is the breed of a dog, which looks like a fox and has light blue eyes, called? Rohit Akut, love and respect animals be friendly have had lots of different pets..
  • Loss: MultiVectorMultipleNegativesRankingLoss with these parameters:
    {
    "scale": 1.0,
    "similarity_fct": "colbert_scores",
    "mini_batch_size": null,
    "score_mini_batch_size": null,
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 128
  • max_steps: 10000
  • learning_rate: 1e-05
  • warmup_steps: 0.05
  • weight_decay: 0.01
  • bf16: True
  • disable_tqdm: True
  • per_device_eval_batch_size: 32
  • load_best_model_at_end: True
  • seed: 12
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • per_device_train_batch_size: 128
  • num_train_epochs: 3.0
  • max_steps: 10000
  • learning_rate: 1e-05
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: None
  • warmup_steps: 0.05
  • optim: adamw_torch_fused
  • optim_args: None
  • weight_decay: 0.01
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • optim_target_modules: None
  • gradient_accumulation_steps: 1
  • average_tokens_across_devices: True
  • max_grad_norm: 1.0
  • label_smoothing_factor: 0.0
  • bf16: True
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • use_liger_kernel: False
  • liger_kernel_config: None
  • use_cache: False
  • neftune_noise_alpha: None
  • torch_empty_cache_steps: None
  • auto_find_batch_size: False
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • include_num_input_tokens_seen: no
  • log_level: passive
  • log_level_replica: warning
  • disable_tqdm: True
  • project: huggingface
  • trackio_space_id: None
  • trackio_bucket_id: None
  • trackio_static_space_id: None
  • per_device_eval_batch_size: 32
  • prediction_loss_only: True
  • eval_on_start: False
  • eval_do_concat_batches: True
  • eval_use_gather_object: False
  • eval_accumulation_steps: None
  • include_for_metrics: []
  • batch_eval_metrics: False
  • save_only_model: False
  • save_on_each_node: False
  • enable_jit_checkpoint: False
  • push_to_hub: False
  • hub_private_repo: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_always_push: False
  • hub_revision: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • restore_callback_states_from_checkpoint: False
  • full_determinism: False
  • seed: 12
  • data_seed: None
  • use_cpu: False
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • dataloader_prefetch_factor: None
  • dataloader_multiprocessing_context: None
  • dataloader_in_order: True
  • remove_unused_columns: True
  • label_names: None
  • train_sampling_strategy: random
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • ddp_static_graph: None
  • ddp_backend: None
  • ddp_timeout: 1800
  • fsdp: None
  • fsdp_config: None
  • deepspeed: None
  • debug: []
  • skip_memory_metrics: True
  • do_predict: False
  • resume_from_checkpoint: None
  • local_rank: -1
  • prompts: None
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}
  • warmup_ratio: None
  • max_length: None

Training Logs

Click to expand
Epoch Step Training Loss Validation Loss NanoMSMARCO_maxsim_ndcg@10 NanoNQ_maxsim_ndcg@10 NanoFiQA2018_maxsim_ndcg@10 NanoBEIR_mean_maxsim_ndcg@10 NanoClimateFEVER_maxsim_ndcg@10 NanoDBPedia_maxsim_ndcg@10 NanoFEVER_maxsim_ndcg@10 NanoHotpotQA_maxsim_ndcg@10 NanoNFCorpus_maxsim_ndcg@10 NanoQuoraRetrieval_maxsim_ndcg@10 NanoSCIDOCS_maxsim_ndcg@10 NanoArguAna_maxsim_ndcg@10 NanoSciFact_maxsim_ndcg@10 NanoTouche2020_maxsim_ndcg@10
-1 0 - - 0.4328 0.2992 0.2658 0.3999 0.1431 0.3914 0.6310 0.5766 0.2569 0.7976 0.2029 0.2863 0.5060 0.4096
0.0003 1 1.7377 - - - - - - - - - - - - - - -
0.0127 50 1.7342 - - - - - - - - - - - - - - -
0.0255 100 1.7067 - - - - - - - - - - - - - - -
0.0382 150 1.6564 - - - - - - - - - - - - - - -
0.0510 200 1.6343 - - - - - - - - - - - - - - -
0.0637 250 1.6082 - - - - - - - - - - - - - - -
0.0765 300 1.5952 - - - - - - - - - - - - - - -
0.0892 350 1.5606 - - - - - - - - - - - - - - -
0.1020 400 1.5329 - - - - - - - - - - - - - - -
0.1147 450 1.4977 - - - - - - - - - - - - - - -
0.1275 500 1.5205 - - - - - - - - - - - - - - -
0.1402 550 1.4602 - - - - - - - - - - - - - - -
0.1530 600 1.4465 - - - - - - - - - - - - - - -
0.1657 650 1.4251 - - - - - - - - - - - - - - -
0.1785 700 1.3911 - - - - - - - - - - - - - - -
0.1912 750 1.3776 - - - - - - - - - - - - - - -
0.2040 800 1.3530 - - - - - - - - - - - - - - -
0.2167 850 1.3297 - - - - - - - - - - - - - - -
0.2295 900 1.3348 - - - - - - - - - - - - - - -
0.2422 950 1.3423 - - - - - - - - - - - - - - -
0.2550 1000 1.2872 - - - - - - - - - - - - - - -
0.2677 1050 1.2880 - - - - - - - - - - - - - - -
0.2805 1100 1.2762 - - - - - - - - - - - - - - -
0.2932 1150 1.2545 - - - - - - - - - - - - - - -
0.3060 1200 1.2463 - - - - - - - - - - - - - - -
0.3187 1250 1.2492 - - - - - - - - - - - - - - -
0.3315 1300 1.2145 - - - - - - - - - - - - - - -
0.3442 1350 1.2369 - - - - - - - - - - - - - - -
0.3570 1400 1.1995 - - - - - - - - - - - - - - -
0.3697 1450 1.1950 - - - - - - - - - - - - - - -
0.3825 1500 1.2016 - - - - - - - - - - - - - - -
0.3952 1550 1.1622 - - - - - - - - - - - - - - -
0.4080 1600 1.1886 - - - - - - - - - - - - - - -
0.4207 1650 1.1866 - - - - - - - - - - - - - - -
0.4335 1700 1.1774 - - - - - - - - - - - - - - -
0.4462 1750 1.1462 - - - - - - - - - - - - - - -
0.4589 1800 1.1655 - - - - - - - - - - - - - - -
0.4717 1850 1.1309 - - - - - - - - - - - - - - -
0.4844 1900 1.1613 - - - - - - - - - - - - - - -
0.4972 1950 1.1465 - - - - - - - - - - - - - - -
0.5099 2000 1.1741 0.8207 0.3460 0.4138 0.2792 0.3463 - - - - - - - - - -
0.5227 2050 1.1287 - - - - - - - - - - - - - - -
0.5354 2100 1.1325 - - - - - - - - - - - - - - -
0.5482 2150 1.1121 - - - - - - - - - - - - - - -
0.5609 2200 1.1046 - - - - - - - - - - - - - - -
0.5737 2250 1.1026 - - - - - - - - - - - - - - -
0.5864 2300 1.1245 - - - - - - - - - - - - - - -
0.5992 2350 1.1253 - - - - - - - - - - - - - - -
0.6119 2400 1.1051 - - - - - - - - - - - - - - -
0.6247 2450 1.0870 - - - - - - - - - - - - - - -
0.6374 2500 1.0775 - - - - - - - - - - - - - - -
0.6502 2550 1.0713 - - - - - - - - - - - - - - -
0.6629 2600 1.0946 - - - - - - - - - - - - - - -
0.6757 2650 1.0682 - - - - - - - - - - - - - - -
0.6884 2700 1.0537 - - - - - - - - - - - - - - -
0.7012 2750 1.0500 - - - - - - - - - - - - - - -
0.7139 2800 1.0920 - - - - - - - - - - - - - - -
0.7267 2850 1.0873 - - - - - - - - - - - - - - -
0.7394 2900 1.0598 - - - - - - - - - - - - - - -
0.7522 2950 1.0745 - - - - - - - - - - - - - - -
0.7649 3000 1.0587 - - - - - - - - - - - - - - -
0.7777 3050 1.0532 - - - - - - - - - - - - - - -
0.7904 3100 1.0609 - - - - - - - - - - - - - - -
0.8032 3150 1.0611 - - - - - - - - - - - - - - -
0.8159 3200 1.0592 - - - - - - - - - - - - - - -
0.8287 3250 1.0413 - - - - - - - - - - - - - - -
0.8414 3300 1.0224 - - - - - - - - - - - - - - -
0.8542 3350 1.0372 - - - - - - - - - - - - - - -
0.8669 3400 1.0544 - - - - - - - - - - - - - - -
0.8797 3450 0.9922 - - - - - - - - - - - - - - -
0.8924 3500 1.0215 - - - - - - - - - - - - - - -
0.9052 3550 1.0052 - - - - - - - - - - - - - - -
0.9179 3600 1.0061 - - - - - - - - - - - - - - -
0.9306 3650 1.0136 - - - - - - - - - - - - - - -
0.9434 3700 1.0236 - - - - - - - - - - - - - - -
0.9561 3750 0.9937 - - - - - - - - - - - - - - -
0.9689 3800 1.0128 - - - - - - - - - - - - - - -
0.9816 3850 1.0399 - - - - - - - - - - - - - - -
0.9944 3900 0.9840 - - - - - - - - - - - - - - -
1.0071 3950 0.9999 - - - - - - - - - - - - - - -
1.0199 4000 1.0204 0.7238 0.3782 0.4405 0.2881 0.3690 - - - - - - - - - -
1.0326 4050 1.0075 - - - - - - - - - - - - - - -
1.0454 4100 1.0111 - - - - - - - - - - - - - - -
1.0581 4150 0.9972 - - - - - - - - - - - - - - -
1.0709 4200 1.0144 - - - - - - - - - - - - - - -
1.0836 4250 1.0054 - - - - - - - - - - - - - - -
1.0964 4300 1.0123 - - - - - - - - - - - - - - -
1.1091 4350 0.9976 - - - - - - - - - - - - - - -
1.1219 4400 0.9798 - - - - - - - - - - - - - - -
1.1346 4450 1.0313 - - - - - - - - - - - - - - -
1.1474 4500 0.9913 - - - - - - - - - - - - - - -
1.1601 4550 0.9911 - - - - - - - - - - - - - - -
1.1729 4600 0.9880 - - - - - - - - - - - - - - -
1.1856 4650 0.9834 - - - - - - - - - - - - - - -
1.1984 4700 0.9489 - - - - - - - - - - - - - - -
1.2111 4750 0.9462 - - - - - - - - - - - - - - -
1.2239 4800 0.9670 - - - - - - - - - - - - - - -
1.2366 4850 0.9766 - - - - - - - - - - - - - - -
1.2494 4900 0.9655 - - - - - - - - - - - - - - -
1.2621 4950 0.9525 - - - - - - - - - - - - - - -
1.2749 5000 0.9818 - - - - - - - - - - - - - - -
1.2876 5050 0.9699 - - - - - - - - - - - - - - -
1.3004 5100 0.9679 - - - - - - - - - - - - - - -
1.3131 5150 0.9810 - - - - - - - - - - - - - - -
1.3259 5200 0.9723 - - - - - - - - - - - - - - -
1.3386 5250 0.9660 - - - - - - - - - - - - - - -
1.3514 5300 0.9688 - - - - - - - - - - - - - - -
1.3641 5350 0.9701 - - - - - - - - - - - - - - -
1.3768 5400 0.9532 - - - - - - - - - - - - - - -
1.3896 5450 0.9817 - - - - - - - - - - - - - - -
1.4023 5500 0.9401 - - - - - - - - - - - - - - -
1.4151 5550 0.9399 - - - - - - - - - - - - - - -
1.4278 5600 0.9505 - - - - - - - - - - - - - - -
1.4406 5650 0.9515 - - - - - - - - - - - - - - -
1.4533 5700 0.9576 - - - - - - - - - - - - - - -
1.4661 5750 0.9672 - - - - - - - - - - - - - - -
1.4788 5800 0.9414 - - - - - - - - - - - - - - -
1.4916 5850 0.9355 - - - - - - - - - - - - - - -
1.5043 5900 0.9615 - - - - - - - - - - - - - - -
1.5171 5950 0.9436 - - - - - - - - - - - - - - -
1.5298 6000 0.9647 0.6846 0.3859 0.4304 0.3025 0.373 - - - - - - - - - -
1.5426 6050 0.9528 - - - - - - - - - - - - - - -
1.5553 6100 0.9359 - - - - - - - - - - - - - - -
1.5681 6150 0.9409 - - - - - - - - - - - - - - -
1.5808 6200 0.9534 - - - - - - - - - - - - - - -
1.5936 6250 0.9594 - - - - - - - - - - - - - - -
1.6063 6300 0.9496 - - - - - - - - - - - - - - -
1.6191 6350 0.9398 - - - - - - - - - - - - - - -
1.6318 6400 0.9368 - - - - - - - - - - - - - - -
1.6446 6450 0.9482 - - - - - - - - - - - - - - -
1.6573 6500 0.9393 - - - - - - - - - - - - - - -
1.6701 6550 0.9601 - - - - - - - - - - - - - - -
1.6828 6600 0.9297 - - - - - - - - - - - - - - -
1.6956 6650 0.9263 - - - - - - - - - - - - - - -
1.7083 6700 0.9391 - - - - - - - - - - - - - - -
1.7211 6750 0.9424 - - - - - - - - - - - - - - -
1.7338 6800 0.9357 - - - - - - - - - - - - - - -
1.7466 6850 0.9470 - - - - - - - - - - - - - - -
1.7593 6900 0.9315 - - - - - - - - - - - - - - -
1.7721 6950 0.9289 - - - - - - - - - - - - - - -
1.7848 7000 0.9206 - - - - - - - - - - - - - - -
1.7976 7050 0.9493 - - - - - - - - - - - - - - -
1.8103 7100 0.9530 - - - - - - - - - - - - - - -
1.8230 7150 0.9379 - - - - - - - - - - - - - - -
1.8358 7200 0.9349 - - - - - - - - - - - - - - -
1.8485 7250 0.9017 - - - - - - - - - - - - - - -
1.8613 7300 0.9407 - - - - - - - - - - - - - - -
1.8740 7350 0.9158 - - - - - - - - - - - - - - -
1.8868 7400 0.9412 - - - - - - - - - - - - - - -
1.8995 7450 0.9240 - - - - - - - - - - - - - - -
1.9123 7500 0.9258 - - - - - - - - - - - - - - -
1.9250 7550 0.9456 - - - - - - - - - - - - - - -
1.9378 7600 0.9428 - - - - - - - - - - - - - - -
1.9505 7650 0.9630 - - - - - - - - - - - - - - -
1.9633 7700 0.9452 - - - - - - - - - - - - - - -
1.9760 7750 0.9222 - - - - - - - - - - - - - - -
1.9888 7800 0.9288 - - - - - - - - - - - - - - -
2.0015 7850 0.9349 - - - - - - - - - - - - - - -
2.0143 7900 0.9355 - - - - - - - - - - - - - - -
2.0270 7950 0.9041 - - - - - - - - - - - - - - -
2.0398 8000 0.9093 0.6678 0.3805 0.4371 0.2995 0.3724 - - - - - - - - - -
2.0525 8050 0.9139 - - - - - - - - - - - - - - -
2.0653 8100 0.9533 - - - - - - - - - - - - - - -
2.0780 8150 0.9314 - - - - - - - - - - - - - - -
2.0908 8200 0.9206 - - - - - - - - - - - - - - -
2.1035 8250 0.9299 - - - - - - - - - - - - - - -
2.1163 8300 0.9174 - - - - - - - - - - - - - - -
2.1290 8350 0.9039 - - - - - - - - - - - - - - -
2.1418 8400 0.9092 - - - - - - - - - - - - - - -
2.1545 8450 0.9086 - - - - - - - - - - - - - - -
2.1673 8500 0.9605 - - - - - - - - - - - - - - -
2.1800 8550 0.9296 - - - - - - - - - - - - - - -
2.1928 8600 0.9196 - - - - - - - - - - - - - - -
2.2055 8650 0.9146 - - - - - - - - - - - - - - -
2.2183 8700 0.9130 - - - - - - - - - - - - - - -
2.2310 8750 0.9037 - - - - - - - - - - - - - - -
2.2438 8800 0.9357 - - - - - - - - - - - - - - -
2.2565 8850 0.9313 - - - - - - - - - - - - - - -
2.2693 8900 0.9179 - - - - - - - - - - - - - - -
2.2820 8950 0.9597 - - - - - - - - - - - - - - -
2.2947 9000 0.9274 - - - - - - - - - - - - - - -
2.3075 9050 0.9092 - - - - - - - - - - - - - - -
2.3202 9100 0.9072 - - - - - - - - - - - - - - -
2.3330 9150 0.9166 - - - - - - - - - - - - - - -
2.3457 9200 0.8974 - - - - - - - - - - - - - - -
2.3585 9250 0.9166 - - - - - - - - - - - - - - -
2.3712 9300 0.9281 - - - - - - - - - - - - - - -
2.3840 9350 0.9292 - - - - - - - - - - - - - - -
2.3967 9400 0.9166 - - - - - - - - - - - - - - -
2.4095 9450 0.9265 - - - - - - - - - - - - - - -
2.4222 9500 0.9142 - - - - - - - - - - - - - - -
2.4350 9550 0.9131 - - - - - - - - - - - - - - -
2.4477 9600 0.8902 - - - - - - - - - - - - - - -
2.4605 9650 0.9464 - - - - - - - - - - - - - - -
2.4732 9700 0.9167 - - - - - - - - - - - - - - -
2.4860 9750 0.9122 - - - - - - - - - - - - - - -
2.4987 9800 0.9217 - - - - - - - - - - - - - - -
2.5115 9850 0.9245 - - - - - - - - - - - - - - -
2.5242 9900 0.8963 - - - - - - - - - - - - - - -
2.5370 9950 0.9256 - - - - - - - - - - - - - - -
2.5497 10000 0.9121 0.6630 0.3796 0.4370 0.3010 0.3725 - - - - - - - - - -
-1 -1 - - 0.3859 0.4300 0.3027 0.4468 0.1919 0.4820 0.6793 0.6840 0.2852 0.8355 0.2217 0.3539 0.5650 0.3920
  • The bold row denotes the saved checkpoint.

Training Time

  • Training: 15.5 minutes
  • Evaluation: 19.4 seconds
  • Total: 15.9 minutes

Framework Versions

  • Python: 3.11.6
  • Sentence Transformers: 6.1.0.dev0
  • Transformers: 5.16.1
  • PyTorch: 2.10.0+cu128
  • Accelerate: 1.14.0
  • Datasets: 4.8.4
  • Tokenizers: 0.23.2

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultiVectorMultipleNegativesRankingLoss 

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
-
Safetensors
Model size
4.39M params
Tensor type
F32
·
Inference Providers NEW
Feature Extraction
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for multi-vector-encoder-testing/bert-tiny-multi-vector

Base model

prajjwal1/bert-tiny
Finetuned
(100)
this model

Dataset used to train multi-vector-encoder-testing/bert-tiny-multi-vector

Papers for multi-vector-encoder-testing/bert-tiny-multi-vector

Evaluation results