README.md

---
base_model: Alibaba-NLP/gte-large-en-v1.5
datasets: []
language:
- en
library_name: sentence-transformers
license: apache-2.0
metrics:
- cosine_accuracy@1
- cosine_accuracy@3
- cosine_accuracy@5
- cosine_accuracy@10
- cosine_precision@1
- cosine_precision@3
- cosine_precision@5
- cosine_precision@10
- cosine_recall@1
- cosine_recall@3
- cosine_recall@5
- cosine_recall@10
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:4275
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
widget:
- source_sentence: The fundamental elements of Goldman Sachs’ robust risk culture
    include governance, risk identification, measurement, mitigation, culture and
    conduct, and infrastructure. They believe these elements work together to complement
    and reinforce each other to produce a comprehensive view of risk management.
  sentences:
  - What are the financial highlights for Bank of America Corp. in its latest fiscal
    year report?
  - What is Berkshire Hathaway's involvement in the energy sector?
  - What is Goldman Sach’s approach towards maintaining a robust risk culture?
- source_sentence: HealthTech Inc.'s new drug for diabetes treatment, launched in
    2021, contributed to approximately 30% of its total revenues for that year.
  sentences:
  - What is IBM's debt to equity ratio as of 2022?
  - In what way does HealthTech Inc's new drug contribute to its revenue generation?
  - What is the revenue breakdown of Alphabet for the year 2021?
- source_sentence: The driving factor behind Tesla’s 2023 growth was the surge in
    demand for electric vehicles.
  sentences:
  - Why did McDonald's observe a decrease in overall revenue in 2023 relative to 2022?
  - What key strategy did Walmart employ to boost its sales in 2016?
  - What was the driving factor behind Tesla's growth in 2023?
- source_sentence: Pfizer is committed to ensuring that people around the world have
    access to its medical products. In line with this commitment, Pfizer has implemented
    programs such as donation drives, price reduction initiatives, and patient assistance
    programs to aid those in need. Furthermore, through partnerships with NGOs and
    governments, Pfizer strives to strengthen healthcare systems in underprivileged
    regions.
  sentences:
  - What is the strategy of Pfizer to improve access to medicines in underprivileged
    areas?
  - What percentage of growth in revenue did Adobe Systems report in June 2020?
  - How is Citigroup differentiating itself among other banks?
- source_sentence: JP Morgan reported total deposits of $2.6 trillion in the year
    ending December 31, 2023.
  sentences:
  - In the fiscal year 2023, what impact did the acquisition of T-Mobile bring to
    the revenue of AT&T?
  - What is the primary source of revenue for the software company, Microsoft?
  - What were JP Morgan's total deposits in 2023?
model-index:
- name: gte-large-en-v1.5-financial-rag-matryoshka
  results:
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 1024
      type: dim_1024
    metrics:
    - type: cosine_accuracy@1
      value: 0.88
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.96
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9866666666666667
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9955555555555555
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.88
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.32
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.19733333333333336
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09955555555555556
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.88
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.96
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9866666666666667
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9955555555555555
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9426916896167131
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.9251851851851851
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.925362962962963
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 768
      type: dim_768
    metrics:
    - type: cosine_accuracy@1
      value: 0.88
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.96
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9866666666666667
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9911111111111112
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.88
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.32
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.19733333333333336
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09911111111111114
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.88
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.96
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9866666666666667
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9911111111111112
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.940825047039427
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.924
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.9245274971941638
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 512
      type: dim_512
    metrics:
    - type: cosine_accuracy@1
      value: 0.8711111111111111
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.96
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9866666666666667
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9911111111111112
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8711111111111111
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.32
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.19733333333333336
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09911111111111114
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8711111111111111
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.96
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9866666666666667
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9911111111111112
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.938126332642602
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.9202962962962962
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.9207248677248678
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 256
      type: dim_256
    metrics:
    - type: cosine_accuracy@1
      value: 0.8755555555555555
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.96
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9866666666666667
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9911111111111112
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8755555555555555
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.32
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.19733333333333336
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09911111111111114
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8755555555555555
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.96
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9866666666666667
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9911111111111112
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9395718726230007
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.9222962962962963
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.9227724867724867
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 128
      type: dim_128
    metrics:
    - type: cosine_accuracy@1
      value: 0.8666666666666667
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9555555555555556
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9866666666666667
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9911111111111112
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8666666666666667
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.3185185185185185
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.19733333333333336
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09911111111111114
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8666666666666667
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9555555555555556
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9866666666666667
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9911111111111112
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9346269584282435
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.9157037037037037
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.9160403095943067
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 64
      type: dim_64
    metrics:
    - type: cosine_accuracy@1
      value: 0.8311111111111111
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.96
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9733333333333334
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9911111111111112
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8311111111111111
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.32
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.19466666666666665
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09911111111111114
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8311111111111111
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.96
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9733333333333334
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9911111111111112
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9208110890988729
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.8971957671957672
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.8975242479721762
      name: Cosine Map@100
---

# financial-rag-matryoshka

Model finetuned for financial use-cases from [Alibaba-NLP/gte-large-en-v1.5](https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5). It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

This model strives to excel tremendously in Financial Document Retrieval Tasks, concurrently preserving a maximum level of generalized performance.
## Model Details

### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [Alibaba-NLP/gte-large-en-v1.5](https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5) <!-- at revision a0d6174973604c8ef416d9f6ed0f4c17ab32d78d -->
- **Maximum Sequence Length:** 8192 tokens
- **Output Dimensionality:** 1024 tokens
- **Similarity Function:** Cosine Similarity
<!-- - **Training Dataset:** Unknown -->
- **Language:** en
- **License:** apache-2.0

### Model Sources

- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)

### Full Model Architecture

```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: NewModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
```

## Usage

### Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

```bash
pip install -U sentence-transformers
```

Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("rbhatia46/gte-large-en-v1.5-financial-rag-matryoshka")
# Run inference
sentences = [
    'JP Morgan reported total deposits of $2.6 trillion in the year ending December 31, 2023.',
    "What were JP Morgan's total deposits in 2023?",
    'What is the primary source of revenue for the software company, Microsoft?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```

<!--
### Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details>
-->

<!--
### Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details>
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

## Evaluation

### Metrics

#### Information Retrieval
* Dataset: `dim_1024`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.88       |
| cosine_accuracy@3   | 0.96       |
| cosine_accuracy@5   | 0.9867     |
| cosine_accuracy@10  | 0.9956     |
| cosine_precision@1  | 0.88       |
| cosine_precision@3  | 0.32       |
| cosine_precision@5  | 0.1973     |
| cosine_precision@10 | 0.0996     |
| cosine_recall@1     | 0.88       |
| cosine_recall@3     | 0.96       |
| cosine_recall@5     | 0.9867     |
| cosine_recall@10    | 0.9956     |
| cosine_ndcg@10      | 0.9427     |
| cosine_mrr@10       | 0.9252     |
| **cosine_map@100**  | **0.9254** |

#### Information Retrieval
* Dataset: `dim_768`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.88       |
| cosine_accuracy@3   | 0.96       |
| cosine_accuracy@5   | 0.9867     |
| cosine_accuracy@10  | 0.9911     |
| cosine_precision@1  | 0.88       |
| cosine_precision@3  | 0.32       |
| cosine_precision@5  | 0.1973     |
| cosine_precision@10 | 0.0991     |
| cosine_recall@1     | 0.88       |
| cosine_recall@3     | 0.96       |
| cosine_recall@5     | 0.9867     |
| cosine_recall@10    | 0.9911     |
| cosine_ndcg@10      | 0.9408     |
| cosine_mrr@10       | 0.924      |
| **cosine_map@100**  | **0.9245** |

#### Information Retrieval
* Dataset: `dim_512`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.8711     |
| cosine_accuracy@3   | 0.96       |
| cosine_accuracy@5   | 0.9867     |
| cosine_accuracy@10  | 0.9911     |
| cosine_precision@1  | 0.8711     |
| cosine_precision@3  | 0.32       |
| cosine_precision@5  | 0.1973     |
| cosine_precision@10 | 0.0991     |
| cosine_recall@1     | 0.8711     |
| cosine_recall@3     | 0.96       |
| cosine_recall@5     | 0.9867     |
| cosine_recall@10    | 0.9911     |
| cosine_ndcg@10      | 0.9381     |
| cosine_mrr@10       | 0.9203     |
| **cosine_map@100**  | **0.9207** |

#### Information Retrieval
* Dataset: `dim_256`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.8756     |
| cosine_accuracy@3   | 0.96       |
| cosine_accuracy@5   | 0.9867     |
| cosine_accuracy@10  | 0.9911     |
| cosine_precision@1  | 0.8756     |
| cosine_precision@3  | 0.32       |
| cosine_precision@5  | 0.1973     |
| cosine_precision@10 | 0.0991     |
| cosine_recall@1     | 0.8756     |
| cosine_recall@3     | 0.96       |
| cosine_recall@5     | 0.9867     |
| cosine_recall@10    | 0.9911     |
| cosine_ndcg@10      | 0.9396     |
| cosine_mrr@10       | 0.9223     |
| **cosine_map@100**  | **0.9228** |

#### Information Retrieval
* Dataset: `dim_128`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value     |
|:--------------------|:----------|
| cosine_accuracy@1   | 0.8667    |
| cosine_accuracy@3   | 0.9556    |
| cosine_accuracy@5   | 0.9867    |
| cosine_accuracy@10  | 0.9911    |
| cosine_precision@1  | 0.8667    |
| cosine_precision@3  | 0.3185    |
| cosine_precision@5  | 0.1973    |
| cosine_precision@10 | 0.0991    |
| cosine_recall@1     | 0.8667    |
| cosine_recall@3     | 0.9556    |
| cosine_recall@5     | 0.9867    |
| cosine_recall@10    | 0.9911    |
| cosine_ndcg@10      | 0.9346    |
| cosine_mrr@10       | 0.9157    |
| **cosine_map@100**  | **0.916** |

#### Information Retrieval
* Dataset: `dim_64`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.8311     |
| cosine_accuracy@3   | 0.96       |
| cosine_accuracy@5   | 0.9733     |
| cosine_accuracy@10  | 0.9911     |
| cosine_precision@1  | 0.8311     |
| cosine_precision@3  | 0.32       |
| cosine_precision@5  | 0.1947     |
| cosine_precision@10 | 0.0991     |
| cosine_recall@1     | 0.8311     |
| cosine_recall@3     | 0.96       |
| cosine_recall@5     | 0.9733     |
| cosine_recall@10    | 0.9911     |
| cosine_ndcg@10      | 0.9208     |
| cosine_mrr@10       | 0.8972     |
| **cosine_map@100**  | **0.8975** |

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Dataset

#### Unnamed Dataset


* Size: 4,275 training samples
* Columns: <code>positive</code> and <code>anchor</code>
* Approximate statistics based on the first 1000 samples:
  |         | positive                                                                            | anchor                                                                            |
  |:--------|:------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|
  | type    | string                                                                              | string                                                                            |
  | details | <ul><li>min: 15 tokens</li><li>mean: 44.74 tokens</li><li>max: 114 tokens</li></ul> | <ul><li>min: 9 tokens</li><li>mean: 18.12 tokens</li><li>max: 32 tokens</li></ul> |
* Samples:
  | positive                                                                                                                                                                                                                                                                         | anchor                                                                                         |
  |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------|
  | <code>At the end of fiscal year 2023, Exxon Mobil reported a debt-to-equity ratio of 0.32, implying that the company used more equity than debt in its capital structure.</code>                                                                                                 | <code>What was the debt-to-equity ratio for Exxon Mobil at the end of fiscal year 2023?</code> |
  | <code>Amazon Web Services (AWS) generated $12.7 billion in net sales in the fourth quarter of 2020, up 28% from the same period of the previous year. It accounted for about 10% of Amazon’s total net sales for the quarter.</code>                                             | <code>How did Amazon's AWS segment perform in the fourth quarter of 2020?</code>               |
  | <code>JPMorgan Chase generates revenues by providing a wide range of banking and financial services. These include investment banking (M&As, advisory), consumer and community banking (home mortgages, auto loans), commercial banking, and asset and wealth management.</code> | <code>What are the key revenue sources for JPMorgan Chase?</code>                              |
* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
  ```json
  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  ```

### Training Hyperparameters
#### Non-Default Hyperparameters

- `eval_strategy`: epoch
- `per_device_train_batch_size`: 32
- `per_device_eval_batch_size`: 16
- `gradient_accumulation_steps`: 16
- `learning_rate`: 2e-05
- `num_train_epochs`: 10
- `lr_scheduler_type`: cosine
- `warmup_ratio`: 0.1
- `bf16`: True
- `tf32`: True
- `load_best_model_at_end`: True
- `optim`: adamw_torch_fused
- `batch_sampler`: no_duplicates

#### All Hyperparameters
<details><summary>Click to expand</summary>

- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: epoch
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 32
- `per_device_eval_batch_size`: 16
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 16
- `eval_accumulation_steps`: None
- `learning_rate`: 2e-05
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1.0
- `num_train_epochs`: 10
- `max_steps`: -1
- `lr_scheduler_type`: cosine
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0.1
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 42
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: True
- `fp16`: False
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: True
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: None
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: True
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch_fused
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: False
- `resume_from_checkpoint`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: False
- `hub_always_push`: False
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`: 
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `dispatch_batches`: None
- `split_batches`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `batch_sampler`: no_duplicates
- `multi_dataset_batch_sampler`: proportional

</details>

### Training Logs
| Epoch   | Step   | Training Loss | dim_1024_cosine_map@100 | dim_128_cosine_map@100 | dim_256_cosine_map@100 | dim_512_cosine_map@100 | dim_64_cosine_map@100 | dim_768_cosine_map@100 |
|:-------:|:------:|:-------------:|:-----------------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|:----------------------:|
| 0.9552  | 8      | -             | 0.9090                  | 0.8848                 | 0.8992                 | 0.9052                 | 0.8775                | 0.9030                 |
| 1.1940  | 10     | 0.4749        | -                       | -                      | -                      | -                      | -                     | -                      |
| 1.9104  | 16     | -             | 0.9170                  | 0.9095                 | 0.9109                 | 0.9201                 | 0.8961                | 0.9212                 |
| 2.3881  | 20     | 0.0862        | -                       | -                      | -                      | -                      | -                     | -                      |
| 2.9851  | 25     | -             | 0.9190                  | 0.9071                 | 0.9160                 | 0.9278                 | 0.8998                | 0.9234                 |
| 3.5821  | 30     | 0.0315        | -                       | -                      | -                      | -                      | -                     | -                      |
| 3.9403  | 33     | -             | 0.9183                  | 0.9053                 | 0.9122                 | 0.9287                 | 0.8998                | 0.9183                 |
| 4.7761  | 40     | 0.0184        | -                       | -                      | -                      | -                      | -                     | -                      |
| 4.8955  | 41     | -             | 0.9225                  | 0.9125                 | 0.9164                 | 0.9260                 | 0.8985                | 0.9220                 |
| 5.9701  | 50     | 0.0135        | 0.9268                  | 0.9132                 | 0.9208                 | 0.9257                 | 0.8961                | 0.9271                 |
| 6.9254  | 58     | -             | 0.9254                  | 0.9158                 | 0.9202                 | 0.9212                 | 0.8938                | 0.9213                 |
| 7.1642  | 60     | 0.0123        | -                       | -                      | -                      | -                      | -                     | -                      |
| **8.0** | **67** | **-**         | **0.9253**              | **0.916**              | **0.9228**             | **0.9207**             | **0.8972**            | **0.9243**             |
| 8.3582  | 70     | 0.01          | -                       | -                      | -                      | -                      | -                     | -                      |
| 8.9552  | 75     | -             | 0.9254                  | 0.9160                 | 0.9213                 | 0.9207                 | 0.9005                | 0.9245                 |
| 9.5522  | 80     | 0.0088        | 0.9254                  | 0.9160                 | 0.9228                 | 0.9207                 | 0.8975                | 0.9245                 |

* The bold row denotes the saved checkpoint.

### Framework Versions
- Python: 3.10.6
- Sentence Transformers: 3.0.1
- Transformers: 4.41.2
- PyTorch: 2.1.2+cu121
- Accelerate: 0.32.1
- Datasets: 2.19.1
- Tokenizers: 0.19.1

## Citation

### BibTeX

#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}
```

#### MatryoshkaLoss
```bibtex
@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
```

#### MultipleNegativesRankingLoss
```bibtex
@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->