jaredmerlo/jared-adjusted

BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("jaredmerlo/jared-adjusted")
# Run inference
sentences = [
    'ategorize\nModel\nMetrics\nAcc Prec Rec\nF1\nBERT-base-multilingual 0.95 0.96 0.94 0.95\nTable 1: Results of the Query Classifier.\n15 tasks based on whether they provide suffi-\ncient information, with specific tasks and exam-\nples illustrated in Figure 2. For tasks entirely\nbased on user-given information, we denote as\n“sufficient”, which need not retrieval; otherwise,\nwe denote as “insufficient”, and retrieval may\nbe necessary. We train a classifier to automate\nthis decision-making process. Experimental de-\ntails are presented in Appendix A.1. Section 4\nexplores the impact of query classification on the workflow, comparing scenarios with and without\nclassification.\n4\nEmbedding Model\nnamespace-Pt/msmarco\nMRR@1 MRR@10 MRR@100\nR@1\nR@10 R@100\nBAAI/LLM-Embedder [20]\n24.79\n37.58\n38.62\n24.07\n66.45\n90.75\nBAAI/bge-base-en-v1.5 [12]\n23.34\n35.80\n36.94\n22.63\n64.12\n90.13\nBAAI/bge-small-en-v1.5 [12]\n23.27\n35.78\n36.89\n22.65\n63.92\n89.80\nBAAI/bge-large-en-v1.5 [12]\n24.63\n37.48\n38.59\n23.91\n65.57\n90.60\nBAAI/b',
    ' 15 tasks based on whether they provide suffi-cient information, with specific tasks and exam-type exam-ples illustrated in Figure 2 . For tasks entirely.given information, we denote as “sufficient” and “insufficient”, which need not retrieval . We train a classifier to automate the decision-making process .',
    ' An open source embedding model is three times smaller than that of BAAI/bge-large-en [12] The size of the two databases is comparable to that of the latter . We select an appropriate vector database for our research based on several key criptions .',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 11 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 11 samples:

  	positive	anchor
  type	string	string
  details	min: 161 tokens mean: 240.0 tokens max: 400 tokens	min: 53 tokens mean: 65.73 tokens max: 85 tokens

• Samples:

  positive	anchor
  cy by precomputing and storing the likelihood of query terms, ranking documents based on their sum. 2.3 Retriever and Generator Fine-tuning Fine-tuning within the RAG framework is crucial for optimizing both retrievers and generators. Some research focuses on fine-tuning the generator to better utilize retriever context [30–32], ensuring faithful and robust generated content. Others fine-tune the retriever to learn to retrieve beneficial passages for the generator [33–35]. Holistic approaches treat RAG as an integrated system, fine-tuning both retriever and generator together to enhance overall performance [36–38], despite increased complexity and integration challenges. Several surveys have extensively discussed current RAG systems, covering aspects like text genera- tion [7, 8], integration with LLMs [6, 39], multimodal [40], and AI-generated content [41]. While these surveys provide comprehensive overviews of existing RAG methodologies, selecting the appro- 3 Which city will the nex	Some research focuses on fine-tuning the generator to better utilize retriever context [30–32] Others fine-tune the retriever to learn to retrieve beneficial passages for the generator [33–35] Holistic approaches treat RAG as an integrated system .
  t World Cup be held? < Search > "French.Washington played a crucial role in the American Revolutionary War, leading the Continental Army against the British. " Please continue writing the above paragraph. < Continuation writing > Background Knowledge "To be, or not to be, that is the question." Please translate this sentence into French. < Translation > Insufficient information Sufficient information Please give me a plan for holding a graduation party. < Planning > If you're currently a computer science student and your computer system encounters a malfunction, what should you do? < Role-play > Write an article about the geography of Europe, focusing on the changes in rainfall in the western part of the country. < Writing > No Retrieval Needed Need to Retrieval Please find a novel that is as famou	Washington played a crucial role in the American                Revolutionary War, leading the                Continental Army against the British. "Please continue writing the above paragraph . Write an article about the geography of Europe, focusing . on the changes in rainfall in the western part of the . western part . of the southeastern Europe. If you're currently a computer science student and your computer system encounters a malfunction, what should . you do?
  s as "One Hundred Years of Solitude". < Search > "Dave is attending his aunt's brother funeral today." Paraphrase the given information effectively. < Rewriting > "The Renaissance was a cultural transformation in European history, marking the revival of arts, sciences, and humanistic thought. The fervor of artists and scholars propelled prosperity and innovation in arts, literature, and science." Give me a summary. < Summarization > Identify who is football players: Messi, Jordan, Kobe. < Closed QA > Tom has three sisters, and each sister has a brother. How many siblings are there in total? < Reasonning > Q: 3,1 A: 3 Q: 2,5 A: 5 Q: 5,7 A: ? < In-context learning > "ChatGPT is a product of OpenAI." Please provide the ownership relationship. < Information extraction > No Background Knowledge If I want to travel from Los Angeles to New York an	"ChatGPT" is a product of "OpenAI" and is based on the open-source knowledge of ChatGPT. s as "One Hundred Years of Solitude" The Renaissance was a cultural transformation in  European history, marking the  revival of arts, sciences, and  humanistic thought. Give me a summary of the Renaissance .

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 8
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• fp16: True
• tf32: False
• optim: adamw_torch_fused
• gradient_checkpointing: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 8
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Framework Versions

• Python: 3.11.12
• Sentence Transformers: 3.1.1
• Transformers: 4.40.2
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 2.19.1
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}