Augmented SBERT : Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks (2021)

Abstract: There are two approaches for pairwise sentence scoring: Cross-encoders, which perform full-attention over the input pair, and Bi-encoders, which map each input independently to a dense vector space. While cross-encoders often achieve higher performance, they are too slow for many practical use cases. Bi-encoders, on the other hand, require substantial training data and fine-tuning over the target task to achieve competitive performance. We present a simple yet efficient data augmentation strategy called Augmented SBERT, where we use the cross-encoder to label a larger set of input pairs to augment the training data for the bi-encoder. We show that, in this process, selecting the sentence pairs is non-trivial and crucial for the success of the method. We evaluate our approach on multiple tasks (in-domain) as well as on a domain adaptation task. Augmented SBERT achieves an improvement of up to 6 points for in-domain and of up to 37 points for domain adaptation tasks compared to the original bi-encoder performance.

## introduction pairwise sentence scoring tasks have wide applications in nlp. they can be used in information retrieval, question answering, duplicate question detection, or clustering. an approach that sets new state-of-the-art performance for many tasks including pairwise sentence scoring is bert @xcite . both sentences are passed to the network and attention is applied across all tokens of the inputs. this approach, where both sentences are simultaneously passed to the network, is called cross-encoder @xcite . a downside of cross-encoders is the extreme computational overhead for many tasks. for example, clustering of 10,000 sentences has a quadratic complexity with a cross-encoder and would require 1 code available: www.sbert.net about 65 hours with bert @xcite . end-to-end information retrieval is also not possible with cross-encoders, as they do not yield independent representations for the inputs that could be indexed. in contrast, bi-encoders such as sentence bert (sbert) @xcite encode each sentence independently and map them to a dense vector space. this allows efficient indexing and comparison. for example, the complexity of clustering 10,000 sentences is reduced from 65 hours to about 5 seconds @xcite . many realworld applications hence depend on the quality of bi-encoders. a drawback of the sbert bi-encoder is usually a lower performance in comparison with the bert cross-encoder. we depict this in figure 1 , where we compare a fine-tuned cross-encoder (bert) and a fine-tuned bi-encoder (sbert) over the popular english sts benchmark dataset foot_0 @xcite for different training sizes and spearman rank correlation (ρ) on the test split. this performance gap is the largest when little training data is available. the bert cross-encoder can compare both inputs simultaneously, while the sbert bi-encoder has to solve the much more challenging task of mapping inputs independently to a meaningful vector space which requires a sufficient amount of training examples for fine-tuning. in this work, we present a data augmentation method, which we call augmented sbert (augs-bert), that uses a bert cross-encoder to improve the performance for the sbert bi-encoder. we use the cross-encoder to label new input pairs, which are added to the training set for the bi-encoder. the sbert bi-encoder is then fine-tuned on this larger augmented training set, which yields a significant performance increase. as we show, selecting the input pairs for soft-labeling with the cross-encoder is non-trivial and crucial for improving performance. our method is easy to apply to many pair classification and regression problems, as we show in the exhaustive evaluation of our approach. first, we evaluate the proposed augsbert method on four diverse tasks: argument similarity, semantic textual similarity, duplicate question detection, and news paraphrase identification. we observe consistent performance increases of 1 to 6 percentage points over the state of the art sbert bi-encoder's performance. next, we demonstrate the strength of augsbert in a domain adaptation scenario. since the bi-encoder is not able to map the new domain to a sensible vector space, the performance drop on the target domain for sbert bi-encoders is much higher than for bert crossencoders. in this scenario, augsbert achieves a performance increase of up to 37 percentage points. ## related work sentence embeddings are a well studied area in recent literature. earlier techniques included unsupervised methods such as skip-thought vectors @xcite and supervised methods such as infersent @xcite or use @xcite . for pairwise scoring tasks, more recent sentence embedding techniques are also able to encode a pair of sentences jointly. among these, bert @xcite can be used as a cross-encoder. both inputs are separated by a special sep token and multi-head attention is applied over all input tokens. while the bert cross-encoder achieves high performances for many sentence pair-tasks, a drawback is that no independent sentence representations are generated. this drawback was ad-dressed by sbert @xcite , which applies bert independently on the inputs followed by mean pooling on the output to create fixed-sized sentence embeddings. @xcite showed that cross-encoders typically outperform bi-encoders on sentence scoring tasks. they proposed a third strategy (poly-encoders), that is in-between cross-and biencoders. poly-encoders utilize two separate transformers, one for the candidate and one for the context. a given candidate is represented by one vector, while the context is jointly encoded with the candidates (similar to cross-encoders). unlike cross-encoder's full self attention technique, polyencoders apply attention between two inputs only at the top layer. poly-encoders have the drawback that they are only practical for certain applications: the score function is not symmetric, i.e., they cannot be applied for tasks with a symmetric similarity relation. further, poly-encoder representations cannot be efficiently indexed, causing issues for retrieval tasks with large corpora sizes. @xcite propose the dipair architecture which, similar to our work, also uses a crossencoder model to annotate unlabeled pairs for finetuning a bi-encoder model. dipair focuses on inference speed and provides a detailed ablation for optimal bi-encoder architectures for performance versus speed trade-offs. the focus of our work are sampling techniques, which we find crucial for performance boosts in the bi-encoder model while keeping its architecture constant. our proposed data augmentation approach is based on semi-supervision @xcite for in-domain tasks, which has been applied successfully for a wide range of tasks. @xcite train a svm model with few gold samples and apply semi-supervision with pre-training neural networks. another common strategy is to generate paraphrases of existent sentences, for example, by replacing words with synonyms @xcite , by using round-trip translation @xcite , or with seq2seq-models @xcite . other approaches generate synthetic data by using generative adversarial networks (tanaka and aranha, 2019), by using a language model to replace certain words @xcite or to generate complete sentences @xcite . these data augmentation approaches have in common that they were applied to single sentence classification tasks. in our work, we focus on sentence pair tasks, for which we need to generate suitable sentence pairs. as we show, randomly combining sentences is insufficient. sampling appropriate pairs has a decisive impact on performance which corresponds to recent findings on similar datasets @xcite . ## methods in this section we present augmented sbert for diverse sentence pair in-domain tasks. we also evaluate our method for domain adaptation tasks. ## datasets sentence pair scoring can be differentiated in regression and classification tasks. regression tasks assign a score to indicate the similarity between the inputs. for classification tasks, we have distinct labels, for example, paraphrase vs. non-paraphrase. ## experimental setup we conduct our experiments using pytorch huggingface's transformers @xcite and the sentence-transformers framework foot_5 (reimers and gurevych, 2019). the latter showed that bert outperforms other transformer-like networks when used as bi-encoder. for english datasets, we use bert-base-uncased and for the spanish dataset we use bert-base-multilingual-cased. every augs-bert model exhibits computational speeds identical to the sbert model @xcite . ## conclusion we presented a simple, yet effective data augmentation approach called augsbert to improve biencoders for pairwise sentence scoring tasks. the idea is based on using a more powerful crossencoder to soft-label new sentence pairs and to include these into the training set. we saw a performance improvement of up to 6 points for in-domain experiments. however, selecting the right sentence pairs for soft-labeling is crucial and the naive approach of randomly selecting pairs fails to achieve a performance gain. we compared several sampling strategies and found that bm25 sampling provides the best trade-off between performance gain and computational complexity. the presented augsbert approach can also be used for domain adaptation, by soft-labeling data on the target domain. in that case, we observe an improvement of up to 37 points compared to an sbert model purely trained on the source domain. yinfei yang, daniel cer, amin ahmad, mandy guo, jax law, noah constant, gustavo hernandez abrego, steve yuan, chris tar, yun-hsuan sung, et al. 2019. multilingual universal sentence encoder for semantic retrieval. arxiv preprint arxiv:1907.04307. adams wei yu, david dohan, minh-thang luong, rui zhao, kai chen, mohammad norouzi, and quoc v. le. 2018. qanet: combining local convolution with global self-attention for reading comprehension. in international conference on learning representations.