nreimers commited on
Commit
8431def
1 Parent(s): 43b322b
README.md ADDED
@@ -0,0 +1,92 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Sentence Embeddings Models trained on Duplicate Questions
2
+ This model is from the [sentence-transformers](https://github.com/UKPLab/sentence-transformers)-repository. It was trained on the [Quora Duplicate Questions dataset](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs). Further details on SBERT can be found in the paper: [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084)
3
+
4
+ For more details, see: [SBERT.net - Pretrained Models](https://www.sbert.net/docs/pretrained_models.html)
5
+
6
+ This model is the multilingual version of quora-distilbert-base, trained on parallel data for 50+ languages.
7
+
8
+ ## Usage (HuggingFace Models Repository)
9
+
10
+ You can use the model directly from the model repository to compute sentence embeddings:
11
+ ```python
12
+ from transformers import AutoTokenizer, AutoModel
13
+ import torch
14
+
15
+
16
+ #Mean Pooling - Take attention mask into account for correct averaging
17
+ def mean_pooling(model_output, attention_mask):
18
+ token_embeddings = model_output[0] #First element of model_output contains all token embeddings
19
+ input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
20
+ sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
21
+ sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
22
+ return sum_embeddings / sum_mask
23
+
24
+
25
+
26
+ #Sentences we want sentence embeddings for
27
+ sentences = ['This framework generates embeddings for each input sentence',
28
+ 'Sentences are passed as a list of string.',
29
+ 'The quick brown fox jumps over the lazy dog.']
30
+
31
+ #Load AutoModel from huggingface model repository
32
+ tokenizer = AutoTokenizer.from_pretrained("model_name")
33
+ model = AutoModel.from_pretrained("model_name")
34
+
35
+ #Tokenize sentences
36
+ encoded_input = tokenizer(sentences, padding=True, truncation=True, max_length=128, return_tensors='pt')
37
+
38
+ #Compute token embeddings
39
+ with torch.no_grad():
40
+ model_output = model(**encoded_input)
41
+
42
+ #Perform pooling. In this case, mean pooling
43
+ sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
44
+ ```
45
+
46
+ ## Usage (Sentence-Transformers)
47
+ Using this model becomes more convenient when you have [sentence-transformers](https://github.com/UKPLab/sentence-transformers) installed:
48
+ ```
49
+ pip install -U sentence-transformers
50
+ ```
51
+
52
+ Then you can use the model like this:
53
+ ```python
54
+ from sentence_transformers import SentenceTransformer
55
+ model = SentenceTransformer('model_name')
56
+ sentences = ['This framework generates embeddings for each input sentence',
57
+ 'Sentences are passed as a list of string.',
58
+ 'The quick brown fox jumps over the lazy dog.']
59
+ sentence_embeddings = model.encode(sentences)
60
+
61
+ print("Sentence embeddings:")
62
+ print(sentence_embeddings)
63
+ ```
64
+
65
+
66
+ ## Citing & Authors
67
+ If you find this model helpful, feel free to cite our publication [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084):
68
+ ```
69
+ @inproceedings{reimers-2019-sentence-bert,
70
+ title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
71
+ author = "Reimers, Nils and Gurevych, Iryna",
72
+ booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
73
+ month = "11",
74
+ year = "2019",
75
+ publisher = "Association for Computational Linguistics",
76
+ url = "http://arxiv.org/abs/1908.10084",
77
+ }
78
+ ```
79
+
80
+ and for the multilingual models:
81
+ If you find this model helpful, feel free to cite our publication [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://arxiv.org/abs/2004.09813):
82
+ ```
83
+ @inproceedings{reimers-2020-multilingual-sentence-bert,
84
+ title = "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation",
85
+ author = "Reimers, Nils and Gurevych, Iryna",
86
+ booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
87
+ month = "11",
88
+ year = "2020",
89
+ publisher = "Association for Computational Linguistics",
90
+ url = "https://arxiv.org/abs/2004.09813",
91
+ }
92
+ ```
config.json ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "activation": "gelu",
3
+ "architectures": [
4
+ "DistilBertModel"
5
+ ],
6
+ "attention_dropout": 0.1,
7
+ "dim": 768,
8
+ "dropout": 0.1,
9
+ "hidden_dim": 3072,
10
+ "initializer_range": 0.02,
11
+ "max_position_embeddings": 512,
12
+ "model_type": "distilbert",
13
+ "n_heads": 12,
14
+ "n_layers": 6,
15
+ "output_past": true,
16
+ "pad_token_id": 0,
17
+ "qa_dropout": 0.1,
18
+ "seq_classif_dropout": 0.2,
19
+ "sinusoidal_pos_embds": false,
20
+ "tie_weights_": true,
21
+ "vocab_size": 119547
22
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:66863c27df8fe2c66905a1c19be5848e4963f8b8d9193577b21e45b9449e453b
3
+ size 538975987
sentence_bert_config.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ {
2
+ "max_seq_length": 128
3
+ }
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"do_lower_case": false, "model_max_length": 512}
vocab.txt ADDED
The diff for this file is too large to render. See raw diff