nreimers commited on
Commit
2aec043
1 Parent(s): d19ebb7
README.md ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Sentence Embeddings Models trained on Duplicate Questions
2
+ This model is from the [sentence-transformers](https://github.com/UKPLab/sentence-transformers)-repository. It was trained on the [Quora Duplicate Questions dataset](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs). Further details on SBERT can be found in the paper: [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084)
3
+
4
+ For more details, see: [SBERT.net - Pretrained Models](https://www.sbert.net/docs/pretrained_models.html)
5
+
6
+ ## Usage (HuggingFace Models Repository)
7
+
8
+ You can use the model directly from the model repository to compute sentence embeddings:
9
+ ```python
10
+ from transformers import AutoTokenizer, AutoModel
11
+ import torch
12
+
13
+
14
+ #Mean Pooling - Take attention mask into account for correct averaging
15
+ def mean_pooling(model_output, attention_mask):
16
+ token_embeddings = model_output[0] #First element of model_output contains all token embeddings
17
+ input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
18
+ sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
19
+ sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
20
+ return sum_embeddings / sum_mask
21
+
22
+
23
+
24
+ #Sentences we want sentence embeddings for
25
+ sentences = ['This framework generates embeddings for each input sentence',
26
+ 'Sentences are passed as a list of string.',
27
+ 'The quick brown fox jumps over the lazy dog.']
28
+
29
+ #Load AutoModel from huggingface model repository
30
+ tokenizer = AutoTokenizer.from_pretrained("model_name")
31
+ model = AutoModel.from_pretrained("model_name")
32
+
33
+ #Tokenize sentences
34
+ encoded_input = tokenizer(sentences, padding=True, truncation=True, max_length=128, return_tensors='pt')
35
+
36
+ #Compute token embeddings
37
+ with torch.no_grad():
38
+ model_output = model(**encoded_input)
39
+
40
+ #Perform pooling. In this case, mean pooling
41
+ sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
42
+ ```
43
+
44
+ ## Usage (Sentence-Transformers)
45
+ Using this model becomes more convenient when you have [sentence-transformers](https://github.com/UKPLab/sentence-transformers) installed:
46
+ ```
47
+ pip install -U sentence-transformers
48
+ ```
49
+
50
+ Then you can use the model like this:
51
+ ```python
52
+ from sentence_transformers import SentenceTransformer
53
+ model = SentenceTransformer('model_name')
54
+ sentences = ['This framework generates embeddings for each input sentence',
55
+ 'Sentences are passed as a list of string.',
56
+ 'The quick brown fox jumps over the lazy dog.']
57
+ sentence_embeddings = model.encode(sentences)
58
+
59
+ print("Sentence embeddings:")
60
+ print(sentence_embeddings)
61
+ ```
62
+
63
+
64
+ ## Citing & Authors
65
+ If you find this model helpful, feel free to cite our publication [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084):
66
+ ```
67
+ @inproceedings{reimers-2019-sentence-bert,
68
+ title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
69
+ author = "Reimers, Nils and Gurevych, Iryna",
70
+ booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
71
+ month = "11",
72
+ year = "2019",
73
+ publisher = "Association for Computational Linguistics",
74
+ url = "http://arxiv.org/abs/1908.10084",
75
+ }
76
+ ```
config.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "activation": "gelu",
3
+ "architectures": [
4
+ "DistilBertModel"
5
+ ],
6
+ "attention_dropout": 0.1,
7
+ "dim": 768,
8
+ "dropout": 0.1,
9
+ "hidden_dim": 3072,
10
+ "initializer_range": 0.02,
11
+ "max_position_embeddings": 512,
12
+ "model_type": "distilbert",
13
+ "n_heads": 12,
14
+ "n_layers": 6,
15
+ "pad_token_id": 0,
16
+ "qa_dropout": 0.1,
17
+ "seq_classif_dropout": 0.2,
18
+ "sinusoidal_pos_embds": false,
19
+ "tie_weights_": true,
20
+ "vocab_size": 30522
21
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:68d315c1638882d9c6cd8901ed497a24de830506773274f60f477620002e113d
3
+ size 265473819
sentence_bert_config.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ {
2
+ "max_seq_length": 128
3
+ }
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"do_lower_case": true, "model_max_length": 512, "special_tokens_map_file": "output/training_nli_distilbert-base-uncased-2020-07-22_10-20-15/0_Transformer/special_tokens_map.json", "full_tokenizer_file": null}
vocab.txt ADDED
The diff for this file is too large to render. See raw diff