kokitabayashi/sbert_FinModernBERT_query

SBERT base trained on BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1

This is a sentence-transformers model finetuned from sentence-transformers/bert-base-nli-mean-tokens on the fin_modern_bert-pairs-sec-synthetic-v1 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/bert-base-nli-mean-tokens
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • fin_modern_bert-pairs-sec-synthetic-v1
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("kokitabayashi/sbert_FinModernBERT_query")
# Run inference
queries = [
    'What were the primary drivers for the increase in net income for the year ended December 31, 2021?',
]
documents = [
    'The increase in net income was primarily due to (i) an increase in our operating profit and (ii) a decrease in other expenses, primarily due to foreign currency gains in the current period as compared to foreign currency losses in the prior-year period, partially offset by (a) an increase in depreciation, amortization and accretion expense, (b) an increase in other operating expense, primarily attributable to acquisition related costs associated with the Telxius Acquisition and the CoreSite Acquisition, and (c) an increase in the income tax provision.',
    'In March 2006, AIG borrowed a total of $1.3 billion on an unsecured basis pursuant to loan agreements with third-party banks, of which $700 million remained outstanding on Decem- 70 AIG 2006 Form 10-K American International Group, Inc. and Subsidiaries ber 31, 2006; $500 million was repaid in February 2007, and the balance matures in March 2007.',
    'Total modifications $ 1,017 $ 649 $ 2,655 $ 775 $ 5,120 $ 592',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.7576, -0.0695, -0.0497]])

Evaluation

Metrics

Information Retrieval

• Dataset: finance-eval
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7557
cosine_accuracy@3	0.9043
cosine_accuracy@5	0.937
cosine_accuracy@10	0.9619
cosine_precision@1	0.7557
cosine_precision@3	0.3014
cosine_precision@5	0.1874
cosine_precision@10	0.0962
cosine_recall@1	0.7557
cosine_recall@3	0.9043
cosine_recall@5	0.937
cosine_recall@10	0.9619
cosine_ndcg@10	0.866
cosine_mrr@10	0.8344
cosine_map@100	0.8362

Training Details

Training Dataset

fin_modern_bert-pairs-sec-synthetic-v1

• Dataset: fin_modern_bert-pairs-sec-synthetic-v1
• Size: 44,709 training samples
• Columns: query and positive
• Approximate statistics based on the first 100 samples:

  	query	positive
  type	string	string
  modality	text	text
  details	min: 16 tokens mean: 27.5 tokens max: 44 tokens	min: 10 tokens mean: 60.88 tokens max: 128 tokens

• Samples:

  query	positive
  What was the total share-based compensation expense included in net income attributable to Biogen Inc. for the year ended December 31, 2015?	Share-based compensation expense included in net income attributable to Biogen Inc. $ 140.5 $ 170.2 $ 172.8 (For the Years Ended December 31, 2015 2014 2013)
  What specific tax implications does the company face regarding its investment portfolio if the portfolio interest exemption on U.S. corporate debt is repealed?	The drafts would repeal the portfolio interest exemption on U.S. corporate debt, which could significantly increase tax liabilities on our investment portfolio or cause us to increase investment in non-U.S. corporate debt.
  What specific organizational changes did the company undertake during fiscal 2013 to support its business initiatives?	Specifically, during fiscal 2013 we undertook organizational changes in order to address major business initiatives, including our desire to accelerate our move to the cloud, transform our customers' experience, increase industry focus to meet customer demands, and develop more effective marketing. These reorganizational efforts included changes to the structure and alignment of our product development and marketing teams and re-organization of our sales teams by industry.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

fin_modern_bert-pairs-sec-synthetic-v1

• Dataset: fin_modern_bert-pairs-sec-synthetic-v1
• Size: 9,580 evaluation samples
• Columns: query and positive
• Approximate statistics based on the first 100 samples:

  	query	positive
  type	string	string
  modality	text	text
  details	min: 16 tokens mean: 27.52 tokens max: 45 tokens	min: 8 tokens mean: 63.29 tokens max: 128 tokens

• Samples:

  query	positive
  What are the potential consequences if the Federal Reserve determines that any of the company's Banks are not 'well-capitalized' or 'well-managed'?	If the Federal Reserve finds that any of our Banks is not “well-capitalized” or “well-managed,” we would be required to enter into an agreement with the Federal Reserve to comply with all applicable capital and management requirements, which may contain additional limitations or conditions relating to our activities.
  What is the concentration of revenue from single customers within the electronic measurement business as of the end of fiscal 2007?	As of the end of fiscal 2007, no single customer represented greater than 10 percent of the net revenue of the business.
  What are the specific requirements for Arch Insurance Europe to maintain its solvency margin in the U.K.?	The FSA requires that Arch Insurance Europe maintain a margin of solvency calculation based on the classes of business for which it is authorized and within its premium income projections applied to its worldwide general business.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• num_train_epochs: 1
• warmup_steps: 0.1
• fp16: True
• per_device_eval_batch_size: 64
• dataloader_num_workers: 2
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 64
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 64
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 2
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

Epoch	Step	Training Loss	Validation Loss	finance-eval_cosine_ndcg@10
-1	-1	-	-	0.2789
0.1431	100	0.4771	0.1072	0.7990
0.2861	200	0.1062	0.0731	0.8299
0.4292	300	0.0877	0.0570	0.8443
0.5722	400	0.0674	0.0488	0.8506
0.7153	500	0.0598	0.0436	0.8599
0.8584	600	0.0487	0.0404	0.8635
1.0	699	-	0.0384	0.8660
-1	-1	-	-	0.8660

Training Time

• Training: 2.0 minutes

Framework Versions

• Python: 3.13.16
• Sentence Transformers: 5.7.0
• Transformers: 5.21.0
• PyTorch: 2.11.0+cu130
• Accelerate: 1.15.0
• Datasets: 4.8.5
• Tokenizers: 0.23.3

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}