Hugging Face's logo

gavinqiangli
/
bge-base-financial-matryoshka

Sentence Similarity
sentence-transformers
Safetensors
English
bert
feature-extraction
Generated from Trainer
dataset_size:6300
loss:MatryoshkaLoss
loss:MultipleNegativesRankingLoss
Eval Results
text-embeddings-inference
Inference Endpoints
Model card Files Community
bge-base-financial-matryoshka / README.md
gavinqiangli's picture
gavinqiangli
Add new SentenceTransformer model
431c4f1 verified
preview code
|
raw
history blame contribute delete
29.5 kB
metadata
base_model: BAAI/bge-base-en-v1.5
language:
  - en
library_name: sentence-transformers
license: apache-2.0
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:6300
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
widget:
  - source_sentence: >-
      Net carrying amount | 10,953 | Less short-term portion | (1,250) | Total
      long-term portion | $ | 9,703
    sentences:
      - How much did restructuring costs amount to in the financial statement?
      - >-
        How much long-term debt remains after accounting for the short-term
        portion as of January 29, 2023?
      - What are the company's environmental sustainability strategies?
  - source_sentence: >-
      Total gross margin for 2023: $169,148 million, for 2022: $170,782 million,
      and for 2021: $152,836 million.
    sentences:
      - How did the total gross margin for Apple Inc. change from 2022 to 2023?
      - >-
        What was the change in noninterest expense for Bank of America from 2022
        to 2023?
      - >-
        How much did FS net revenue increase by in fiscal 2023 compared to
        fiscal 2022?
  - source_sentence: >-
      Goldman Sachs manages its activities in three business segments: Global
      Banking & Markets, Asset & Wealth Management, and Platform Solutions.
    sentences:
      - >-
        What are the three business segments of Goldman Sachs as mentioned in
        their 2023 Form 10-K?
      - How are financial statement indexes presented in a document?
      - >-
        What was the total foreign currency transaction loss recorded for the
        year ended December 31, 2023?
  - source_sentence: NIKE, Inc. was incorporated in 1967 under the laws of the State of Oregon.
    sentences:
      - What was the global gender equity status at Meta in July 2023?
      - When was NIKE, Inc. incorporated and under the laws of which state?
      - How is product warranty liability estimated by the company?
  - source_sentence: >-
      In 2023, total assets associated with derivatives designated as hedging
      instruments amounted to $1,527 million, while total liabilities amounted
      to $5,962 million.
    sentences:
      - How are delivery sales categorized in financial statements?
      - >-
        What was the balance of deferred net loss on derivatives included in
        accumulated other comprehensive income as of December 31, 2023?
      - >-
        What was the total value of assets and liabilities associated with
        derivatives designated as hedging instruments in 2023?
model-index:
  - name: BGE base Financial Matryoshka
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 768
          type: dim_768
        metrics:
          - type: cosine_accuracy@1
            value: 0.7314285714285714
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8414285714285714
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.88
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9171428571428571
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7314285714285714
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.28047619047619043
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.17599999999999996
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09171428571428569
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.7314285714285714
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8414285714285714
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.88
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9171428571428571
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8242643635674787
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7945634920634922
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7974204140430639
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 512
          type: dim_512
        metrics:
          - type: cosine_accuracy@1
            value: 0.73
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8442857142857143
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8757142857142857
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9114285714285715
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.73
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.2814285714285714
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.1751428571428571
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09114285714285712
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.73
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8442857142857143
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8757142857142857
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9114285714285715
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8208470282419681
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7917534013605444
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7950732633962434
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 256
          type: dim_256
        metrics:
          - type: cosine_accuracy@1
            value: 0.7242857142857143
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8285714285714286
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8671428571428571
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9057142857142857
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7242857142857143
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.27619047619047615
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.1734285714285714
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09057142857142855
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.7242857142857143
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8285714285714286
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8671428571428571
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9057142857142857
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8144984416133947
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7853690476190478
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7887014688628511
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 128
          type: dim_128
        metrics:
          - type: cosine_accuracy@1
            value: 0.7042857142857143
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8128571428571428
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8557142857142858
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9057142857142857
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7042857142857143
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.2709523809523809
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.17114285714285712
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09057142857142855
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.7042857142857143
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8128571428571428
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8557142857142858
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9057142857142857
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8018796849794548
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.769049886621315
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7722252928484385
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 64
          type: dim_64
        metrics:
          - type: cosine_accuracy@1
            value: 0.6657142857142857
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.78
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.82
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.8685714285714285
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.6657142857142857
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.26
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.16399999999999998
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.08685714285714284
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.6657142857142857
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.78
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.82
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.8685714285714285
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.7667584555431229
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7344319727891154
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7397691471615258
            name: Cosine Map@100

BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5 on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: BAAI/bge-base-en-v1.5
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 768 tokens
  • Similarity Function: Cosine Similarity
  • Training Dataset:
    • json
  • Language: en
  • License: apache-2.0

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("gavinqiangli/bge-base-financial-matryoshka")
# Run inference
sentences = [
    'In 2023, total assets associated with derivatives designated as hedging instruments amounted to $1,527 million, while total liabilities amounted to $5,962 million.',
    'What was the total value of assets and liabilities associated with derivatives designated as hedging instruments in 2023?',
    'What was the balance of deferred net loss on derivatives included in accumulated other comprehensive income as of December 31, 2023?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Metric Value
cosine_accuracy@1 0.7314
cosine_accuracy@3 0.8414
cosine_accuracy@5 0.88
cosine_accuracy@10 0.9171
cosine_precision@1 0.7314
cosine_precision@3 0.2805
cosine_precision@5 0.176
cosine_precision@10 0.0917
cosine_recall@1 0.7314
cosine_recall@3 0.8414
cosine_recall@5 0.88
cosine_recall@10 0.9171
cosine_ndcg@10 0.8243
cosine_mrr@10 0.7946
cosine_map@100 0.7974

Information Retrieval

Metric Value
cosine_accuracy@1 0.73
cosine_accuracy@3 0.8443
cosine_accuracy@5 0.8757
cosine_accuracy@10 0.9114
cosine_precision@1 0.73
cosine_precision@3 0.2814
cosine_precision@5 0.1751
cosine_precision@10 0.0911
cosine_recall@1 0.73
cosine_recall@3 0.8443
cosine_recall@5 0.8757
cosine_recall@10 0.9114
cosine_ndcg@10 0.8208
cosine_mrr@10 0.7918
cosine_map@100 0.7951

Information Retrieval

Metric Value
cosine_accuracy@1 0.7243
cosine_accuracy@3 0.8286
cosine_accuracy@5 0.8671
cosine_accuracy@10 0.9057
cosine_precision@1 0.7243
cosine_precision@3 0.2762
cosine_precision@5 0.1734
cosine_precision@10 0.0906
cosine_recall@1 0.7243
cosine_recall@3 0.8286
cosine_recall@5 0.8671
cosine_recall@10 0.9057
cosine_ndcg@10 0.8145
cosine_mrr@10 0.7854
cosine_map@100 0.7887

Information Retrieval

Metric Value
cosine_accuracy@1 0.7043
cosine_accuracy@3 0.8129
cosine_accuracy@5 0.8557
cosine_accuracy@10 0.9057
cosine_precision@1 0.7043
cosine_precision@3 0.271
cosine_precision@5 0.1711
cosine_precision@10 0.0906
cosine_recall@1 0.7043
cosine_recall@3 0.8129
cosine_recall@5 0.8557
cosine_recall@10 0.9057
cosine_ndcg@10 0.8019
cosine_mrr@10 0.769
cosine_map@100 0.7722

Information Retrieval

Metric Value
cosine_accuracy@1 0.6657
cosine_accuracy@3 0.78
cosine_accuracy@5 0.82
cosine_accuracy@10 0.8686
cosine_precision@1 0.6657
cosine_precision@3 0.26
cosine_precision@5 0.164
cosine_precision@10 0.0869
cosine_recall@1 0.6657
cosine_recall@3 0.78
cosine_recall@5 0.82
cosine_recall@10 0.8686
cosine_ndcg@10 0.7668
cosine_mrr@10 0.7344
cosine_map@100 0.7398

Training Details

Training Dataset

json

  • Dataset: json
  • Size: 6,300 training samples
  • Columns: positive and anchor
  • Approximate statistics based on the first 1000 samples:
    positive anchor
    type string string
    details
    • min: 10 tokens
    • mean: 46.04 tokens
    • max: 289 tokens
    • min: 2 tokens
    • mean: 20.38 tokens
    • max: 42 tokens
  • Samples:
    positive anchor
    The Nominating and Corporate Governance Committee of our Board of Directors is responsible for reviewing and discussing with management our practices related to ESG. What is the role of the Nominating and Corporate Governance Committee at NVIDIA?
    Deferred tax assets and deferred tax liabilities included in the Consolidated Balance Sheets as follows: As of October 31, 2023: Deferred tax assets were $3,155 million and Deferred tax liabilities were $44 million. As of October 31, 2022: Deferred tax assets were $2,167 million and Deferred tax liabilities were $121 million. The total net deferred tax assets were $3,111 million in 2023 and $2,046 million in 2022. What was the change in HP's net deferred tax assets from 2022 to 2023?
    Sales and marketing expense increased $247 million, or 16%, in 2023, compared to 2022, primarily due to a $177 million increase in marketing activities associated with our marketing campaigns and launches and our search engine marketing and advertising spend. What was the major reason for the increase in Sales and Marketing expenses in 2023?
  • Loss: MatryoshkaLoss with these parameters:
    {
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: epoch
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 16
  • gradient_accumulation_steps: 16
  • learning_rate: 2e-05
  • num_train_epochs: 4
  • lr_scheduler_type: cosine
  • warmup_ratio: 0.1
  • fp16: True
  • load_best_model_at_end: True
  • optim: adamw_torch_fused
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: epoch
  • prediction_loss_only: True
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 16
  • eval_accumulation_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 4
  • max_steps: -1
  • lr_scheduler_type: cosine
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: True
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss dim_768_cosine_map@100 dim_512_cosine_map@100 dim_256_cosine_map@100 dim_128_cosine_map@100 dim_64_cosine_map@100
0.8122 10 1.5963 - - - - -
0.9746 12 - 0.7791 0.7824 0.7662 0.7483 0.7086
1.6244 20 0.6846 - - - - -
1.9492 24 - 0.7924 0.7903 0.7859 0.7664 0.7327
2.4365 30 0.4956 - - - - -
2.9239 36 - 0.7962 0.7939 0.7886 0.7716 0.7378
3.2487 40 0.3998 - - - - -
3.8985 48 - 0.7974 0.7951 0.7887 0.7722 0.7398
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.10.12
  • Sentence Transformers: 3.2.1
  • Transformers: 4.41.2
  • PyTorch: 2.1.2+cu121
  • Accelerate: 0.34.2
  • Datasets: 2.19.1
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss 

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss 

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}