Add new SentenceTransformer model.

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

2_Dense/config.json

@@ -0,0 +1 @@
+{"in_features": 768, "out_features": 768, "bias": false, "activation_function": "torch.nn.modules.linear.Identity"}

2_Dense/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5365f1d15b18a7ae7d2e500bf404b6d3cf45397afe1a2086641ff56bc5d7d1a
+size 2360171

README.md

@@ -1,95 +1,69 @@
-# Sentence Embedding Model for MS MARCO Passage Retrieval
+---
+pipeline_tag: sentence-similarity
+tags:
+- sentence-transformers
+- feature-extraction
+- sentence-similarity
+- transformers
+- transformers
+- transformers
+- transformers
+- transformers
+- transformers
+- transformers
+- transformers
+---
 
+# sentence-transformers/msmarco-distilbert-base-dot-prod-v3
 
-This a `distilroberta-base` model from the [sentence-transformers](https://github.com/UKPLab/sentence-transformers)-repository. It was trained on the [MS MARCO Passage Retrieval dataset](https://github.com/microsoft/MSMARCO-Passage-Ranking): Given a search query, it finds the relevant passages.
+This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.
 
-You can use this model for semantic search. Details can be found on: [SBERT.net - Semantic Search](https://www.sbert.net/examples/applications/semantic-search/README.html).
 
-This model was optimized to be used with **dot-product** as similarity function between queries and documents.
 
+## Usage (Sentence-Transformers)
 
-## Training
-
-Details about the training of the models can be found here: [SBERT.net - MS MARCO](https://www.sbert.net/examples/training/ms_marco/README.html)
-
-## Performance
+Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
 
-For performance details, see: [SBERT.net - Pre-Trained Models - MS MARCO](https://www.sbert.net/docs/pretrained-models/msmarco-v3.html)
+```
+pip install -U sentence-transformers
+```
 
-## Usage (HuggingFace Models Repository)
+Then you can use the model like this:
 
-You can use the model directly from the model repository to compute sentence embeddings:
 ```python
-from transformers import AutoTokenizer, AutoModel
-import torch
-
-
-#Mean Pooling - Take attention mask into account for correct averaging
-def mean_pooling(model_output, attention_mask):
-    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
-    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
-    sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
-    sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
-    return sum_embeddings / sum_mask
+from sentence_transformers import SentenceTransformer
+sentences = ["This is an example sentence", "Each sentence is converted"]
 
+model = SentenceTransformer('sentence-transformers/msmarco-distilbert-base-dot-prod-v3')
+embeddings = model.encode(sentences)
+print(embeddings)
+```
 
 
-# Queries we want embeddings for
-queries = ['What is the capital of France?', 'How many people live in New York City?']
 
-# Passages that provide answers
-passages = ['Paris is the capital of France', 'New York City is the most populous city in the United States, with an estimated 8,336,817 people living in the city, according to U.S. Census estimates dating July 1, 2019']
+## Evaluation Results
 
-#Load AutoModel from huggingface model repository
-tokenizer = AutoTokenizer.from_pretrained("model_name")
-model = AutoModel.from_pretrained("model_name")
 
-def compute_embeddings(sentences):
-	#Tokenize sentences
-	encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
 
-	#Compute query embeddings
-	with torch.no_grad():
-		model_output = model(**encoded_input)
+For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=sentence-transformers/msmarco-distilbert-base-dot-prod-v3)
 
-	#Perform pooling. In this case, mean pooling
-	return mean_pooling(model_output, encoded_input['attention_mask'])
 
-query_embeddings = compute_embeddings(queries)
-passage_embeddings = compute_embeddings(passages)
-```
 
-## Usage (Sentence-Transformers)
-Using this model becomes more convenient when you have [sentence-transformers](https://github.com/UKPLab/sentence-transformers) installed:
+## Full Model Architecture
 ```
-pip install -U sentence-transformers
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: DistilBertModel 
+  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
+  (2): Dense({'in_features': 768, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
+)
 ```
 
-Then you can use the model like this:
-```python
-from sentence_transformers import SentenceTransformer
-model = SentenceTransformer('model_name')
-
-# Queries we want embeddings for
-queries = ['What is the capital of France?', 'How many people live in New York City?']
-
-# Passages that provide answers
-passages = ['Paris is the capital of France', 'New York City is the most populous city in the United States, with an estimated 8,336,817 people living in the city, according to U.S. Census estimates dating July 1, 2019']
-
-query_embeddings = model.encode(queries)
-passage_embeddings = model.encode(passages)
-```
-
-## Changes in v3
-The models from v2 have been used for find for all training queries similar passages. An [MS MARCO Cross-Encoder](ce-msmarco.md) based on the electra-base-model has been then used to classify if these retrieved passages answer the question.
-
-If they received a low score by the cross-encoder, we saved them as hard negatives: They got a high score from the bi-encoder, but a low-score from the (better) cross-encoder.
-
-We then trained the v2 models with these new hard negatives.
-
 ## Citing & Authors
+
+This model was trained by [sentence-transformers](https://www.sbert.net/). 
+        
 If you find this model helpful, feel free to cite our publication [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084):
-``` 
+```bibtex 
 @inproceedings{reimers-2019-sentence-bert,
     title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
     author = "Reimers, Nils and Gurevych, Iryna",

config.json

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "../output/distilbert-base-uncased-mined_hard_neg-mean-pooling-dot_prod-no_identifier-epoch10-batchsize75-2021-03-21_13-53-07/0_Transformer",
+  "_name_or_path": "old_models/msmarco-distilbert-base-dot-prod-v3/0_Transformer",
   "activation": "gelu",
   "architectures": [
     "DistilBertModel"
@@ -18,6 +18,6 @@
   "seq_classif_dropout": 0.2,
   "sinusoidal_pos_embds": false,
   "tie_weights_": true,
-  "transformers_version": "4.4.1",
+  "transformers_version": "4.7.0",
   "vocab_size": 30522
 }

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.0.0",
+    "transformers": "4.7.0",
+    "pytorch": "1.9.0+cu102"
+  }
+}

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Dense",
+    "type": "sentence_transformers.models.Dense"
+  }
+]

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0d2805041081f6c6e0bd3c903a6c2b5a23551ec8063b91bcee90d05478cdaa71
-size 265491187
+oid sha256:e67d89134719a423d5978c8760b259f4e3106dc6b25c6bac4ccb50e7fbbeda38
+size 265486777

tokenizer_config.json

@@ -1 +1 @@
-{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "special_tokens_map_file": null, "name_or_path": "../output/distilbert-base-uncased-mined_hard_neg-mean-pooling-dot_prod-no_identifier-epoch10-batchsize75-2021-03-21_13-53-07/0_Transformer", "do_basic_tokenize": true, "never_split": null}
+{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "special_tokens_map_file": null, "name_or_path": "old_models/msmarco-distilbert-base-dot-prod-v3/0_Transformer", "do_basic_tokenize": true, "never_split": null}