Add new SentenceTransformer model.

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

README.md

@@ -1,18 +1,42 @@
-# Sentence Embedding Model for MS MARCO Passage Retrieval
+---
+pipeline_tag: sentence-similarity
+tags:
+- sentence-transformers
+- feature-extraction
+- sentence-similarity
+- transformers
+---
 
+# sentence-transformers/msmarco-distilroberta-base-v2
 
-This a `distilroberta-base` model from the [sentence-transformers](https://github.com/UKPLab/sentence-transformers)-repository. It was trained on the [MS MARCO Passage Retrieval dataset](https://github.com/microsoft/MSMARCO-Passage-Ranking): Given a search query, it finds the relevant passages.
+This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.
 
-You can use this model for semantic search. Details can be found on: [SBERT.net - Semantic Search](https://www.sbert.net/examples/applications/semantic-search/README.html) and [SBERT.net - Information Retrieval](https://www.sbert.net/examples/applications/information-retrieval/README.html) 
 
 
-## Training
+## Usage (Sentence-Transformers)
+
+Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+
+```
+pip install -U sentence-transformers
+```
+
+Then you can use the model like this:
+
+```python
+from sentence_transformers import SentenceTransformer
+sentences = ["This is an example sentence", "Each sentence is converted"]
+
+model = SentenceTransformer('sentence-transformers/msmarco-distilroberta-base-v2')
+embeddings = model.encode(sentences)
+print(embeddings)
+```
+
 
-Details about the training of the models can be found here: [SBERT.net - MS MARCO](https://www.sbert.net/examples/training/ms_marco/README.html)
 
-## Usage (HuggingFace Models Repository)
+## Usage (HuggingFace Transformers)
+Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
 
-You can use the model directly from the model repository to compute sentence embeddings:
 ```python
 from transformers import AutoTokenizer, AutoModel
 import torch
@@ -22,62 +46,54 @@ import torch
 def mean_pooling(model_output, attention_mask):
     token_embeddings = model_output[0] #First element of model_output contains all token embeddings
     input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
-    sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
-    sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
-    return sum_embeddings / sum_mask
+    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
 
 
+# Sentences we want sentence embeddings for
+sentences = ['This is an example sentence', 'Each sentence is converted']
 
-# Queries we want embeddings for
-queries = ['What is the capital of France?', 'How many people live in New York City?']
+# Load model from HuggingFace Hub
+tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/msmarco-distilroberta-base-v2')
+model = AutoModel.from_pretrained('sentence-transformers/msmarco-distilroberta-base-v2')
 
-# Passages that provide answers
-passages = ['Paris is the capital of France', 'New York City is the most populous city in the United States, with an estimated 8,336,817 people living in the city, according to U.S. Census estimates dating July 1, 2019']
+# Tokenize sentences
+encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
 
-#Load AutoModel from huggingface model repository
-tokenizer = AutoTokenizer.from_pretrained("model_name")
-model = AutoModel.from_pretrained("model_name")
+# Compute token embeddings
+with torch.no_grad():
+    model_output = model(**encoded_input)
 
-def compute_embeddings(sentences):
-	#Tokenize sentences
-	encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
+# Perform pooling. In this case, max pooling.
+sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
 
-	#Compute query embeddings
-	with torch.no_grad():
-		model_output = model(**encoded_input)
+print("Sentence embeddings:")
+print(sentence_embeddings)
+```
 
-	#Perform pooling. In this case, mean pooling
-	return mean_pooling(model_output, encoded_input['attention_mask'])
 
-query_embeddings = compute_embeddings(queries)
-passage_embeddings = compute_embeddings(passages)
-```
 
-## Usage (Sentence-Transformers)
-Using this model becomes more convenient when you have [sentence-transformers](https://github.com/UKPLab/sentence-transformers) installed:
-```
-pip install -U sentence-transformers
-```
+## Evaluation Results
 
-Then you can use the model like this:
-```python
-from sentence_transformers import SentenceTransformer
-model = SentenceTransformer('model_name')
 
-# Queries we want embeddings for
-queries = ['What is the capital of France?', 'How many people live in New York City?']
 
-# Passages that provide answers
-passages = ['Paris is the capital of France', 'New York City is the most populous city in the United States, with an estimated 8,336,817 people living in the city, according to U.S. Census estimates dating July 1, 2019']
+For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=sentence-transformers/msmarco-distilroberta-base-v2)
 
-query_embeddings = model.encode(queries)
-passage_embeddings = model.encode(passages)
-```
 
 
+## Full Model Architecture
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 350, 'do_lower_case': False}) with Transformer model: RobertaModel 
+  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
+)
+```
+
 ## Citing & Authors
+
+This model was trained by [sentence-transformers](https://www.sbert.net/). 
+        
 If you find this model helpful, feel free to cite our publication [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084):
-``` 
+```bibtex 
 @inproceedings{reimers-2019-sentence-bert,
     title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
     author = "Reimers, Nils and Gurevych, Iryna",

config.json

@@ -1,4 +1,5 @@
 {
+  "_name_or_path": "old_models/msmarco-distilroberta-base-v2/0_Transformer",
   "architectures": [
     "RobertaModel"
   ],
@@ -17,6 +18,9 @@
   "num_attention_heads": 12,
   "num_hidden_layers": 6,
   "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.7.0",
   "type_vocab_size": 1,
+  "use_cache": true,
   "vocab_size": 50265
 }

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.0.0",
+    "transformers": "4.7.0",
+    "pytorch": "1.9.0+cu102"
+  }
+}

merges.txt

@@ -1,4 +1,4 @@
-#version: 0.2
+#version: 0.2 - Trained by `huggingface/tokenizers`
 Ġ t
 Ġ a
 h e

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5585abb96b5be7a4594a4286a251c71ec43fb6d9f9d738831ca1f3b5649429fb
-size 328520407
+oid sha256:287f1deba2b7737731981f72ecba16ad92cea0d8ded916b99ee5808617fa57af
+size 328515953

sentence_bert_config.json

@@ -1,3 +1,4 @@
 {
-  "max_seq_length": 350
+  "max_seq_length": 350,
+  "do_lower_case": false
 }

tokenizer_config.json

@@ -1 +1 @@
-{"model_max_length": 512}
+{"unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "add_prefix_space": false, "errors": "replace", "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "model_max_length": 512, "special_tokens_map_file": "old_models/msmarco-distilroberta-base-v2/0_Transformer/special_tokens_map.json", "name_or_path": "old_models/msmarco-distilroberta-base-v2/0_Transformer"}