michelerondinelli01/mnlp-hw1-biencoder

SentenceTransformer based on distilbert/distilbert-base-uncased

This is a sentence-transformers model finetuned from distilbert/distilbert-base-uncased. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: distilbert/distilbert-base-uncased
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'DistilBertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("michelerondinelli01/mnlp-hw1-biencoder")
# Run inference
sentences = [
    'what is the name that is given to a symbiotic relationship',
    'Symbiosis can vary between mutualism, commensalism, and parasitism, though these grade into each other,. In mutualism, both organisms benefit. In commensalism, one benefits and the other is unaffected; in parasitism, one benefits and the other is harmed.',
    'Some of these relationships are so close that we speak of the composite of two species as one unit; for example, we speak of the composite of algae and fungi as lichens.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6165, 0.2146],
#         [0.6165, 1.0000, 0.4963],
#         [0.2146, 0.4963, 1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: dev-eval
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.4825
cosine_accuracy@3	0.6975
cosine_accuracy@5	0.7712
cosine_accuracy@10	0.8425
cosine_precision@1	0.4825
cosine_precision@3	0.2325
cosine_precision@5	0.1542
cosine_precision@10	0.0842
cosine_recall@1	0.4825
cosine_recall@3	0.6975
cosine_recall@5	0.7712
cosine_recall@10	0.8425
cosine_ndcg@10	0.6642
cosine_mrr@10	0.6068
cosine_map@100	0.6116

Training Details

Training Dataset

Unnamed Dataset

• Size: 14,400 training samples
• Columns: sentence_0, sentence_1, and sentence_2
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	sentence_2
  type	string	string	string
  details	min: 9 tokens mean: 11.88 tokens max: 27 tokens	min: 14 tokens mean: 124.72 tokens max: 455 tokens	min: 3 tokens mean: 101.92 tokens max: 490 tokens

• Samples:

  sentence_0	sentence_1	sentence_2
  what are the predators of the praying mantis	Mantises are preyed on by vertebrates such as frogs, lizards, and birds, and by invertebrates such as spiders, large species of hornets, and ants. Some hunting wasps, such as some species of Tachytes also paralyse some species of mantis to feed their young. Generally, mantises protect themselves by camouflage, most species being cryptically colored to resemble foliage or other backgrounds, both to avoid predators and to better snare their prey. Those that live on uniformly colored surfaces such as bare earth or tree bark are dorsoventrally flattened so as to eliminate shadows that might reveal their presence. The species from different families called flower mantises are aggressive mimics: they resemble flowers convincingly enough to attract prey that come to collect pollen and nectar. Some species in Africa and Australia are able to turn black after a molt towards the end of the dry season; at this time of year, bush fires occur and this coloration enables them to blend in with the fi...	Gardeners who prefer to avoid pesticides may encourage mantises in the hope of controlling insect pests. However, mantises do not have key attributes of biological pest control agents; they do not specialize in a single pest insect, and do not multiply rapidly in response to an increase in such a prey species, but are general predators. They eat whatever they can catch, including both harmful and beneficial insects. They therefore have "negligible value" in biological control.
  what types of structures have the same function but do not share a common ancestry	Convergent evolution is the independent evolution of similar features in species of different lineages. Convergent evolution creates analogous structures that have similar form or function but were not present in the last common ancestor of those groups. The cladistic term for the same phenomenon is homoplasy. The recurrent evolution of flight is a classic example, as flying insects, birds, pterosaurs, and bats have independently evolved the useful capacity of flight. Functionally similar features that have arisen through convergent evolution are analogous, whereas homologous structures or traits have a common origin but can have dissimilar functions. Bird, bat, and pterosaur wings are analogous structures, but their forelimbs are homologous, sharing an ancestral state despite serving different functions.	Birds and bats have homologous limbs because they are both ultimately derived from terrestrial tetrapods, but their flight mechanisms are only analogous, so their wings are examples of functional convergence. The two groups have powered flight, evolved independently. Their wings differ substantially in construction. The bat wing is a membrane stretched across four extremely elongated fingers and the legs. The airfoil of the bird wing is made of feathers, strongly attached to the forearm (the ulna) and the highly fused bones of the wrist and hand (the carpometacarpus), with only tiny remnants of two fingers remaining, each anchoring a single feather. So, while the wings of bats and birds are functionally convergent, they are not anatomically convergent. Similarly, the extinct pterosaur also shows an independent evolution of vertebrate fore- and hindlimbs to wing. An even more distantly related group, the insects, have wings that evolved separately from different organs.
  where does the st. johns river begin and end	Starting in Indian River County and meeting the Atlantic Ocean at Duval County, the St. Johns is Florida's primary commercial and recreational waterway. It flows north from its headwaters, originating in the direction of the Lake Wales Ridge, which is only slightly elevated at 30 feet (9.1 m) above sea level. Because of this low elevation drop, the river has a long backwater. It ebbs and flows with tides that pass through the barrier islands and up the channel. Uniquely, it shares the same regional terrain as the parallel Kissimmee River, although the Kissimmee flows south.	All of Florida's abundant fresh water is the result of precipitation that returns to the atmosphere in a process called evapotranspiration that involves evaporation and transpiration of moisture from plants. As rains fall, most of the water is directed to lakes, streams, and rivers. However, a significant amount of fresh water is held underground but close to the surface in aquifers. A surficial aquifer consisting mostly of clay, shells, and sand is over a confining layer of denser materials. Wells are drilled in the surficial aquifer, which supplies better quality water in areas where the deepest aquifer has a high mineral content. Occasionally the confining layer is fractured to allow breaches of water to percolate down to recharge the layer below. The Floridan Aquifer, underneath the confining layer, underlies the entire state and portions of Georgia, Alabama, and South Carolina. It is particularly accessible in the northern part of Florida, serving as the fresh water source of metr...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• num_train_epochs: 4
• fp16: True
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	dev-eval_cosine_ndcg@10
0.6667	300	-	0.6099
1.0	450	-	0.6425
1.1111	500	0.5279	-
1.3333	600	-	0.6491
2.0	900	-	0.6593
2.2222	1000	0.1084	-
2.6667	1200	-	0.6669
3.0	1350	-	0.6659
3.3333	1500	0.0406	0.6662
4.0	1800	-	0.6642

Training Time

• Training: 43.2 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}