Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity (2020)

Abstract: We address the task of unsupervised Semantic Textual Similarity (STS) by ensembling diverse pre-trained sentence encoders into sentence meta-embeddings. We apply, extend and evaluate different meta-embedding methods from the word embedding literature at the sentence level, including dimensionality reduction (Yin and Schütze, 2016), generalized Canonical Correlation Analysis (Rastogi et al., 2015) and cross-view auto-encoders (Bollegala and Bao, 2018). Our sentence meta-embeddings set a new unsupervised State of The Art (SoTA) on the STS Benchmark and on the STS12-STS16 datasets, with gains of between 3.7% and 6.4% Pearson’s r over single-source systems.

## introduction word meta-embeddings have been shown to exceed single-source word embeddings on word-level semantic benchmarks @xcite . presumably, this is because they combine the complementary strengths of their components. there has been recent interest in pre-trained "universal" sentence encoders, i.e., functions that encode diverse semantic features of sentences into fixed-size vectors @xcite . since these sentence encoders differ in terms of their architecture and training data, we hypothesize that their strengths are also complementary and that they can benefit from meta-embeddings. to test this hypothesis, we adapt different metaembedding methods from the word embedding literature. these include dimensionality reduction @xcite , cross-view autoencoders @xcite and generalized canonical correlation analysis (gcca) @xcite . the latter method was also used by @xcite for domain-specific duplicate question detection. our sentence encoder ensemble includes three models from the recent literature: sentence-bert @xcite , the universal sentence encoder @xcite and averaged paranmt vectors @xcite . our meta-embeddings outperform every one of their constituent single-source embeddings on sts12-16 @xcite and on the sts benchmark @xcite . crucially, since our meta-embeddings are agnostic to the contents of their ensemble, future improvements may be possible by adding new encoders. word meta-embeddings combine ensembles of word embeddings by various operations: yin and schütze (2016) use concatenation, svd and linear projection, coates and @xcite show that averaging word embeddings has properties similar to concatenation. @xcite apply generalized canonical correlation analysis (gcca) to an ensemble of word vectors. @xcite learn word meta-embeddings using autoencoder architectures. neill and bollegala (2018) evaluate different loss functions for autoencoder word meta-embeddings, while @xcite explore locally linear mappings. ## sentence meta-embedding methods below, we assume access to an ensemble of pretrained sentence encoders, denoted f 1 . . . f j . every f j maps from the (infinite) set of possible sentences s to a fixed-size d j -dimensional vector. word meta-embeddings are usually learned from a finite vocabulary of word types @xcite . sentence embeddings lack such a "vocabulary", as they can encode any member of s. therefore, we train on a sample s ⊂ s, i.e., on a corpus of unlabeled sentences. ## conclusion inspired by the success of word meta-embeddings, we have shown how to apply different metaembedding techniques to ensembles of sentence encoders. all sentence meta-embeddings consistently outperform their individual single-source components on the sts benchmark and the sts12-16 datasets, with a new unsupervised sota set by our gcca meta-embeddings. because sentence metaembeddings are agnostic to the size and specifics of their ensemble, it should be possible to add new encoders to the ensemble, potentially improving performance further.