Add new SentenceTransformer model

2_Dense/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d5e0d234a36d4cd513a1f95bf413a6eea8972f8c835c5270a3e57d4eabf1b5ed
-size 1181344
+oid sha256:bc630dbae3594eeb0a6c8575cfa1de738bc5b246dffca9741b2d4f5851dd7989
+size 2362528

3_Dense/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:38a849eb316f199a702045d159bf2fd1eec62ad05f2bc051329e42e986c6731a
-size 1181344
+oid sha256:88ac9eb6f9310d3ee402976ca8daab484eaf9e3dfcdd22f7dc49d36f7e30ed38
+size 2362528

README.md

@@ -63,6 +63,49 @@ datasets:
 - redis/langcache-sentencepairs-v2
 pipeline_tag: sentence-similarity
 library_name: sentence-transformers
+metrics:
+- cosine_accuracy@1
+- cosine_precision@1
+- cosine_recall@1
+- cosine_ndcg@10
+- cosine_mrr@1
+- cosine_map@100
+- cosine_auc_precision_cache_hit_ratio
+- cosine_auc_similarity_distribution
+model-index:
+- name: Redis fine-tuned BiEncoder model for semantic caching on LangCache
+  results:
+  - task:
+      type: custom-information-retrieval
+      name: Custom Information Retrieval
+    dataset:
+      name: test
+      type: test
+    metrics:
+    - type: cosine_accuracy@1
+      value: 0.5880219631236443
+      name: Cosine Accuracy@1
+    - type: cosine_precision@1
+      value: 0.5880219631236443
+      name: Cosine Precision@1
+    - type: cosine_recall@1
+      value: 0.5706780985738924
+      name: Cosine Recall@1
+    - type: cosine_ndcg@10
+      value: 0.7717640552650085
+      name: Cosine Ndcg@10
+    - type: cosine_mrr@1
+      value: 0.5880219631236443
+      name: Cosine Mrr@1
+    - type: cosine_map@100
+      value: 0.7213999116625115
+      name: Cosine Map@100
+    - type: cosine_auc_precision_cache_hit_ratio
+      value: 0.35292771304732773
+      name: Cosine Auc Precision Cache Hit Ratio
+    - type: cosine_auc_similarity_distribution
+      value: 0.1674589579463346
+      name: Cosine Auc Similarity Distribution
 ---
 
 # Redis fine-tuned BiEncoder model for semantic caching on LangCache
@@ -128,9 +171,9 @@ print(embeddings.shape)
 # Get the similarity scores for the embeddings
 similarities = model.similarity(embeddings, embeddings)
 print(similarities)
-# tensor([[1.0000, 1.0000, 0.7693],
-#         [1.0000, 1.0000, 0.7693],
-#         [0.7693, 0.7693, 1.0000]])
+# tensor([[1.0000, 1.0000, 0.5313],
+#         [1.0000, 1.0000, 0.5313],
+#         [0.5313, 0.5313, 1.0000]])
 ```
 
 <!--
@@ -157,6 +200,26 @@ You can finetune this model on your own dataset.
 *List how the model may foreseeably be misused and address what users ought not to do with the model.*
 -->
 
+## Evaluation
+
+### Metrics
+
+#### Custom Information Retrieval
+
+* Dataset: `test`
+* Evaluated with <code>ir_evaluator.CustomInformationRetrievalEvaluator</code>
+
+| Metric                               | Value      |
+|:-------------------------------------|:-----------|
+| cosine_accuracy@1                    | 0.588      |
+| cosine_precision@1                   | 0.588      |
+| cosine_recall@1                      | 0.5707     |
+| **cosine_ndcg@10**                   | **0.7718** |
+| cosine_mrr@1                         | 0.588      |
+| cosine_map@100                       | 0.7214     |
+| cosine_auc_precision_cache_hit_ratio | 0.3529     |
+| cosine_auc_similarity_distribution   | 0.1675     |
+
 <!--
 ## Bias, Risks and Limitations
 
@@ -246,6 +309,7 @@ You can finetune this model on your own dataset.
 - `dataloader_persistent_workers`: True
 - `push_to_hub`: True
 - `hub_model_id`: redis/langcache-embed-v3
+- `eval_on_start`: True
 - `batch_sampler`: no_duplicates
 
 #### All Hyperparameters
@@ -359,7 +423,7 @@ You can finetune this model on your own dataset.
 - `neftune_noise_alpha`: None
 - `optim_target_modules`: None
 - `batch_eval_metrics`: False
-- `eval_on_start`: False
+- `eval_on_start`: True
 - `use_liger_kernel`: False
 - `liger_kernel_config`: None
 - `eval_use_gather_object`: False
@@ -372,6 +436,12 @@ You can finetune this model on your own dataset.
 
 </details>
 
+### Training Logs
+| Epoch | Step | Validation Loss | test_cosine_ndcg@10 |
+|:-----:|:----:|:---------------:|:-------------------:|
+| 0     | 0    | 1.0850          | 0.7718              |
+
+
 ### Framework Versions
 - Python: 3.12.3
 - Sentence Transformers: 5.1.0

config.json

@@ -12,7 +12,7 @@
   "cls_token_id": 50281,
   "decoder_bias": true,
   "deterministic_flash_attn": false,
-  "dtype": "bfloat16",
+  "dtype": "float32",
   "embedding_dropout": 0.0,
   "eos_token_id": 50282,
   "global_attn_every_n_layers": 3,

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:95d02211c4cca89113f9f3e93ed91f5176bf50170faa2cb835f7bfea15bb9dd2
-size 298041696
+oid sha256:04aa7437b7f98ed3f652e300c1d767d07c1864c10b3055ea63831997faefa8d6
+size 596070136