# sentence-transformers /distilbert-base-nli-mean-tokens

nreimers
HF staff
 1 --- 2 pipeline_tag: feature-extraction 3 license: apache-2.0 4 tags: 5 - sentence-transformers 6 - feature-extraction 7 - sentence-similarity 8 - transformers 9 --- 10 11 **⚠️ This model is deprecated. Please don't use it as it produces sentence embeddings of low quality. You can find recommended sentence embedding models here: [SBERT.net - Pretrained Models](https://www.sbert.net/docs/pretrained_models.html)** 12 13 14 # sentence-transformers/distilbert-base-nli-mean-tokens 15 16 This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. 17 18 19 20 ## Usage (Sentence-Transformers) 21 22 Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: 23 24  25 pip install -U sentence-transformers 26  27 28 Then you can use the model like this: 29 30 python 31 from sentence_transformers import SentenceTransformer 32 sentences = ["This is an example sentence", "Each sentence is converted"] 33 34 model = SentenceTransformer('sentence-transformers/distilbert-base-nli-mean-tokens') 35 embeddings = model.encode(sentences) 36 print(embeddings) 37  38 39 40 41 ## Usage (HuggingFace Transformers) 42 Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings. 43 44 python 45 from transformers import AutoTokenizer, AutoModel 46 import torch 47 48 49 #Mean Pooling - Take attention mask into account for correct averaging 50 def mean_pooling(model_output, attention_mask): 51 token_embeddings = model_output[0] #First element of model_output contains all token embeddings 52 input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float() 53 return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9) 54 55 56 # Sentences we want sentence embeddings for 57 sentences = ['This is an example sentence', 'Each sentence is converted'] 58 59 # Load model from HuggingFace Hub 60 tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/distilbert-base-nli-mean-tokens') 61 model = AutoModel.from_pretrained('sentence-transformers/distilbert-base-nli-mean-tokens') 62 63 # Tokenize sentences 64 encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt') 65 66 # Compute token embeddings 67 with torch.no_grad(): 68 model_output = model(**encoded_input) 69 70 # Perform pooling. In this case, max pooling. 71 sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask']) 72 73 print("Sentence embeddings:") 74 print(sentence_embeddings) 75  76 77 78 79 ## Evaluation Results 80 81 82 83 For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=sentence-transformers/distilbert-base-nli-mean-tokens) 84 85 86 87 ## Full Model Architecture 88  89 SentenceTransformer( 90 (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: DistilBertModel 91 (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False}) 92 ) 93  94 95 ## Citing & Authors 96 97 This model was trained by [sentence-transformers](https://www.sbert.net/). 98 99 If you find this model helpful, feel free to cite our publication [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084): 100 bibtex 101 @inproceedings{reimers-2019-sentence-bert, 102 title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", 103 author = "Reimers, Nils and Gurevych, Iryna", 104 booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", 105 month = "11", 106 year = "2019", 107 publisher = "Association for Computational Linguistics", 108 url = "http://arxiv.org/abs/1908.10084", 109 } 110