julien-c HF staff commited on
Commit
924bf7d
1 Parent(s): 3264ff5

Migrate model card from transformers-repo

Browse files

Read announcement at https://discuss.huggingface.co/t/announcement-all-model-cards-will-be-migrated-to-hf-co-model-repos/2755
Original file history: https://github.com/huggingface/transformers/commits/master/model_cards/sentence-transformers/bert-base-nli-max-tokens/README.md

Files changed (1) hide show
  1. README.md +88 -0
README.md ADDED
@@ -0,0 +1,88 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ tags:
4
+ - exbert
5
+ license: apache-2.0
6
+ datasets:
7
+ - snli
8
+ - multi_nli
9
+ ---
10
+
11
+ # BERT base model (uncased) for Sentence Embeddings
12
+ This is the `bert-base-nli-max-tokens` model from the [sentence-transformers](https://github.com/UKPLab/sentence-transformers)-repository. The sentence-transformers repository allows to train and use Transformer models for generating sentence and text embeddings.
13
+ The model is described in the paper [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084)
14
+
15
+ ## Usage (HuggingFace Models Repository)
16
+
17
+ You can use the model directly from the model repository to compute sentence embeddings. It uses max pooling to generate a fixed sized sentence embedding:
18
+ ```python
19
+ from transformers import AutoTokenizer, AutoModel
20
+ import torch
21
+
22
+
23
+ #Max Pooling - Take the max value over time for every dimension
24
+ def max_pooling(model_output, attention_mask):
25
+ token_embeddings = model_output[0] #First element of model_output contains all token embeddings
26
+ input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
27
+ token_embeddings[input_mask_expanded == 0] = -1e9 # Set padding tokens to large negative value
28
+ max_over_time = torch.max(token_embeddings, 1)[0]
29
+ return max_over_time
30
+
31
+
32
+ #Sentences we want sentence embeddings for
33
+ sentences = ['This framework generates embeddings for each input sentence',
34
+ 'Sentences are passed as a list of string.',
35
+ 'The quick brown fox jumps over the lazy dog.']
36
+
37
+ #Load AutoModel from huggingface model repository
38
+ tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/bert-base-nli-max-tokens")
39
+ model = AutoModel.from_pretrained("sentence-transformers/bert-base-nli-max-tokens")
40
+
41
+ #Tokenize sentences
42
+ encoded_input = tokenizer(sentences, padding=True, truncation=True, max_length=128, return_tensors='pt')
43
+
44
+ #Compute token embeddings
45
+ with torch.no_grad():
46
+ model_output = model(**encoded_input)
47
+
48
+ #Perform pooling. In this case, max pooling
49
+ sentence_embeddings = max_pooling(model_output, encoded_input['attention_mask'])
50
+
51
+
52
+ print("Sentence embeddings:")
53
+ print(sentence_embeddings)
54
+ ```
55
+
56
+ ## Usage (Sentence-Transformers)
57
+ Using this model becomes more convenient when you have [sentence-transformers](https://github.com/UKPLab/sentence-transformers) installed:
58
+ ```
59
+ pip install -U sentence-transformers
60
+ ```
61
+
62
+ Then you can use the model like this:
63
+ ```python
64
+ from sentence_transformers import SentenceTransformer
65
+ model = SentenceTransformer('bert-base-nli-max-tokens')
66
+ sentences = ['This framework generates embeddings for each input sentence',
67
+ 'Sentences are passed as a list of string.',
68
+ 'The quick brown fox jumps over the lazy dog.']
69
+ sentence_embeddings = model.encode(sentences)
70
+
71
+ print("Sentence embeddings:")
72
+ print(sentence_embeddings)
73
+ ```
74
+
75
+
76
+ ## Citing & Authors
77
+ If you find this model helpful, feel free to cite our publication [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084):
78
+ ```
79
+ @inproceedings{reimers-2019-sentence-bert,
80
+ title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
81
+ author = "Reimers, Nils and Gurevych, Iryna",
82
+ booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
83
+ month = "11",
84
+ year = "2019",
85
+ publisher = "Association for Computational Linguistics",
86
+ url = "http://arxiv.org/abs/1908.10084",
87
+ }
88
+ ```