README.md

metadata

tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:6552
  - loss:MultipleNegativesRankingLoss
base_model: BAAI/bge-small-en-v1.5
widget:
  - source_sentence: >-
      What property is denoted as the M→M property in the queueing network
      literature?
    sentences:
      - >-
        The LOFAR system introduces two additional levels in the beam hierarchy:
        the compound (tile) beam and the station beam.
      - >-
        The desired pseudonoise sequence in a CDMA system has the
        characteristics that the fraction of 0's and 1's is almost half-and-half
        over the period, and the shifted versions of the pseudonoise sequence
        are nearly orthogonal to each other. If the shift of the pseudonoise
        sequence is randomized, it becomes a random process.
      - >-
        The M→M property in the queueing network literature denotes the
        independence of individual queues in the long term.
  - source_sentence: >-
      Which type of channel condition has better path loss exponent (PLE) in
      terms of AA (air to air) and AG (air to ground) propagation channels?
    sentences:
      - >-
        The goal of the Fixed Access Information API is to provide access
        network related information for the multitude of fixed access
        technologies.
      - >-
        Error mitigation is a technique to reduce the impact of errors in
        near-term quantum systems without requiring full fault-tolerant quantum
        codes.
      - >-
        From the document, it is mentioned that the AA channel has better
        conditions than the AG channel in terms of path loss exponent (PLE).
  - source_sentence: What is the goal of a functionality extraction attack?
    sentences:
      - >-
        Deep learning can automatically extract high-level features from data,
        reducing the need for manual feature engineering.
      - >-
        The goal of a functionality extraction attack is to create knock-off
        models that mimic the behavior of an existing machine learning model.
      - >-
        The main advantage of using wind turbine towers for communication is
        that they already have a reliable power grid connection.
  - source_sentence: What is MTU?
    sentences:
      - >-
        The worst-case complexity of average consensus is exponential in the
        number of nodes, but it can be reduced to linear if an upper bound on
        the total number of nodes is known.
      - >-
        In a normally clad fiber, at long wavelengths, the MFD is large compared
        to the core diameter and the electric field extends far into the
        cladding region.
      - >-
        MTU (Maximum Transmission Unit) represents the largest size of a data
        packet that can be sent over a network without fragmentation.
  - source_sentence: >-
      What should the AP or PCP do if it is not decentralized AP or PCP
      clustering capable or a decentralized AP or PCP cluster is not present?
    sentences:
      - >-
        When a Data, Management or Extension frame is received, a STA inserts it
        in an appropriate cache.
      - >-
        If the AP or PCP is not decentralized AP or PCP clustering capable or a
        decentralized AP or PCP cluster is not present, it should set its
        Cluster Member Role to 0 (not currently participating in a cluster) and
        remain unclustered.
      - >-
        Analog beamforming based on slowly-varying second order statistics of
        the CSI reduces the dimension of the effective instantaneous CSI for
        digital beamforming within each coherent fading block, which helps to
        relieve the signaling overhead.
datasets:
  - dinho1597/Telecom-QA-MultipleChoice
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_recall@1
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
model-index:
  - name: SentenceTransformer based on BAAI/bge-small-en-v1.5
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: telecom ir eval
          type: telecom-ir-eval
        metrics:
          - type: cosine_accuracy@1
            value: 0.965675057208238
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.992372234935164
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9931350114416476
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9938977879481312
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.965675057208238
            name: Cosine Precision@1
          - type: cosine_recall@1
            value: 0.965675057208238
            name: Cosine Recall@1
          - type: cosine_ndcg@10
            value: 0.9824027787882591
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.9784334023464457
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.9786169716375667
            name: Cosine Map@100

SentenceTransformer based on BAAI/bge-small-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-small-en-v1.5 on the telecom-qa-multiple_choice dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-small-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • telecom-qa-multiple_choice

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'What should the AP or PCP do if it is not decentralized AP or PCP clustering capable or a decentralized AP or PCP cluster is not present?',
    'If the AP or PCP is not decentralized AP or PCP clustering capable or a decentralized AP or PCP cluster is not present, it should set its Cluster Member Role to 0 (not currently participating in a cluster) and remain unclustered.',
    'When a Data, Management or Extension frame is received, a STA inserts it in an appropriate cache.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: telecom-ir-eval
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.9657
cosine_accuracy@3	0.9924
cosine_accuracy@5	0.9931
cosine_accuracy@10	0.9939
cosine_precision@1	0.9657
cosine_recall@1	0.9657
cosine_ndcg@10	0.9824
cosine_mrr@10	0.9784
cosine_map@100	0.9786

Training Details

Training Dataset

telecom-qa-multiple_choice

• Dataset: telecom-qa-multiple_choice at 73aebbb
• Size: 6,552 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 4 tokens mean: 18.95 tokens max: 49 tokens	min: 9 tokens mean: 29.33 tokens max: 112 tokens

• Samples:

  anchor	positive
  What is the goal of a jammer in a mobile edge caching system?	The goal of a jammer in a mobile edge caching system is to interrupt ongoing radio transmissions of the edge node with cached chunks or caching users and prevent access to cached content. Additionally, jammers aim to deplete the resources of edge nodes, caching users, and sensors during failed communication attempts.
  Which type of DRL uses DNNs (Deep Neural Networks) to fit action values and employs experience replay and target networks to ensure stable training convergence?	Value-based DRL, such as Deep Q-Learning (DQL), uses DNNs to fit action values and employs experience replay and target networks to ensure stable training convergence.
  What is the relationship between the curvature of the decision boundary and the robustness of a network?	The lower the curvature of the decision boundaries, the more robust the network.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

telecom-qa-multiple_choice

• Dataset: telecom-qa-multiple_choice at 73aebbb
• Size: 6,552 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 4 tokens mean: 18.87 tokens max: 56 tokens	min: 8 tokens mean: 29.45 tokens max: 91 tokens

• Samples:

  anchor	positive
  Which forward error correction (FEC) codes are available for the THz single carrier mode?	The THz single carrier mode (THz-SC PHY) in the IEEE 802.15.3d standard supports two low-density parity-check (LDPC) codes: 14/15 LDPC (1440,1344) and 11/15 LDPC (1440,1056).
  Which multiple access technique allows users to access the channel simultaneously using the same frequency and time resources, with different power levels?	Non-Orthogonal Multiple Access (NOMA) allows users to access the channel simultaneously using the same frequency and time resources, but with different power levels.
  What is the power gain when doubling the number of antennas?	Doubling the number of antennas yields a 3-dB power gain.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 512
• per_device_eval_batch_size: 512
• weight_decay: 0.01
• num_train_epochs: 15
• lr_scheduler_type: cosine_with_restarts
• warmup_ratio: 0.1
• fp16: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 512
• per_device_eval_batch_size: 512
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 15
• max_steps: -1
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	telecom-ir-eval_cosine_ndcg@10
1.2727	15	1.0332	0.0968	0.9725
2.5455	30	0.2091	0.0518	0.9808
3.8182	45	0.0997	0.0470	0.9824

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.3.1
• Transformers: 4.47.1
• PyTorch: 2.5.1+cu121
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}