kokitabayashi/bge-m3_FinModernBERT_query

bge-m3 base trained on BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1

This is a sentence-transformers model finetuned from BAAI/bge-m3 on the fin_modern_bert-pairs-sec-synthetic-v1 dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • fin_modern_bert-pairs-sec-synthetic-v1
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("kokitabayashi/bge-m3_FinModernBERT_query")
# Run inference
queries = [
    'What is the specific debt-to-capital resources ratio covenant for the company, and what was the actual ratio as of December 31, 2005?',
]
documents = [
    'Our only financial covenant exists with respect to our credit facility and our synthetic lease VIE obligations. The covenant requires that we not exceed a 37.5% debt to capital resources ratio as defined in the agreements. This ratio at December 31, 2005 was 21.5%.',
    'S & P has a stable outlook on all of the ratings of the Companies, except for John Alden Life Insurance Company and Time Insurance Company, which have a negative outlook.',
    'In 2017, production costs increased $10 million, or 3.5%, primarily due to a: • $6 million increase in purchased water principally due to usage increases in our California subsidiary, the result of the lifting in April 2017 of California water usage restrictions that had been mandated in 2015 due to the state’s extreme drought; • $2 million increase in fuel and power largely attributable to higher system delivery of water and supplier price increases in our Missouri and California subsidiaries; and • $2 million increase in waste disposal mainly due to higher sludge removal costs in our Illinois and Missouri subsidiaries.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.8410, 0.0446, 0.0053]])

Evaluation

Metrics

Information Retrieval

• Dataset: finance-eval
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8257
cosine_accuracy@3	0.9508
cosine_accuracy@5	0.9714
cosine_accuracy@10	0.987
cosine_precision@1	0.8257
cosine_precision@3	0.3169
cosine_precision@5	0.1943
cosine_precision@10	0.0987
cosine_recall@1	0.8257
cosine_recall@3	0.9508
cosine_recall@5	0.9714
cosine_recall@10	0.987
cosine_ndcg@10	0.9145
cosine_mrr@10	0.8903
cosine_map@100	0.891

Training Details

Training Dataset

fin_modern_bert-pairs-sec-synthetic-v1

• Dataset: fin_modern_bert-pairs-sec-synthetic-v1
• Size: 44,709 training samples
• Columns: query and positive
• Approximate statistics based on the first 100 samples:

  	query	positive
  type	string	string
  modality	text	text
  details	min: 18 tokens mean: 29.77 tokens max: 49 tokens	min: 10 tokens mean: 68.54 tokens max: 228 tokens

• Samples:

  query	positive
  What is the breakdown of the estimated fair value of debt securities classified as short-term investments by their stated maturities?	The following table summarizes the cost and estimated fair value of debt securities classified as short-term investments based on stated maturities. Due within one year $ 660,921 $ 664,819; Due within two years 268,508 271,841; Due within three years 72,187 73,547; Due after three years 113,446 119,498; Total $ 1,115,062 $ 1,129,705
  What were the specific drivers for the 24% increase in HUMIRA revenues in the United States during 2016?	In the United States, HUMIRA revenues increased 24% in 2016 and 29% in 2015, driven by prescription volume, favorable pricing, market growth across all indications and higher market share.
  What is the average rental rate and physical occupancy for the Avalon at Mission Bay III property in San Francisco?	Avalon at Mission Bay III North San Francisco, CA 260 261,169 2009 1,004 93.1 % 95.2 % 96.2 % 4,445 4.43 147,918

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

fin_modern_bert-pairs-sec-synthetic-v1

• Dataset: fin_modern_bert-pairs-sec-synthetic-v1
• Size: 9,580 evaluation samples
• Columns: query and positive
• Approximate statistics based on the first 100 samples:

  	query	positive
  type	string	string
  modality	text	text
  details	min: 18 tokens mean: 30.81 tokens max: 43 tokens	min: 11 tokens mean: 65.29 tokens max: 171 tokens

• Samples:

  query	positive
  How does the seasonality of orders and revenues typically affect the Life Sciences and Applied Markets business segment?	The life sciences and applied markets business is susceptible to seasonality in its orders and revenues primarily related to U.S. and foreign government budgets, chemical and energy and environmental customers and large pharmaceutical company budgets. Historically, the result is that our first and fourth fiscal quarters tend to deliver the strongest profits for this group.
  What specific factors are identified as potential drivers for increased claim severity in auto bodily injury, property damage, and homeowners coverages?	The following factors may impact claim severity for auto bodily injury, property damage and homeowners coverages: • Bodily injury — inflation in medical costs, litigation trends and precedents and regulation • Vehicle property damage — inflation in repair costs, including parts and labor rates, mix of total losses declared, costs associated with repairing sophisticated newer vehicles, model year and used-car values • Homeowners — inflation in the construction industry, building materials and home furnishings, changes in the mix of loss type, and other economic and environmental factors, including short-term supply imbalances for services and supplies in areas affected by catastrophes
  What is the remaining indebtedness on the subordinated notes as of December 31, 2005, and what were the conversion terms for these notes?	As of December 31, 2005, our remaining indebtedness under the subordinated notes was approximately $75.4 million at maturity, due to conversion of subordinated notes into common stock. Each $1,000 aggregate principal face value subordinated note is convertible at the holder’s option at any time through maturity into 40.404 shares of our common stock at an initial conversion price of $8.36 per share.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• num_train_epochs: 4
• learning_rate: 2e-05
• warmup_steps: 0.1
• bf16: True
• dataloader_num_workers: 2
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 64
• num_train_epochs: 4
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 2
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

Epoch	Step	Training Loss	Validation Loss	finance-eval_cosine_ndcg@10
-1	-1	-	-	0.7985
1.0	699	0.0548	0.0035	0.9021
2.0	1398	0.0166	0.0025	0.9088
3.0	2097	0.0107	0.0028	0.9127
4.0	2796	0.0089	0.0024	0.9145
-1	-1	-	-	0.9145

Training Time

• Training: 14.5 minutes

Framework Versions

• Python: 3.13.16
• Sentence Transformers: 5.7.0
• Transformers: 5.22.0
• PyTorch: 2.11.0+cu130
• Accelerate: 1.15.0
• Datasets: 4.8.5
• Tokenizers: 0.23.3

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}