Contrasting distinct structured views to learn sentence embeddings (2021)

Abstract: We propose a self-supervised method that builds sentence embeddings from the combination of diverse explicit syntactic structures of a sentence. We assume structure is crucial to building consistent representations as we expect sentence meaning to be a function of both syntax and semantic aspects. In this perspective, we hypothesize that some linguistic representations might be better adapted given the considered task or sentence. We, therefore, propose to learn individual representation functions for different syntactic frameworks jointly. Again, by hypothesis, all such functions should encode similar semantic information differently and consequently, be complementary for building better sentential semantic embeddings. To assess such hypothesis, we propose an original contrastive multi-view framework that induces an explicit interaction between models during the training phase. We make experiments combining various structures such as dependency, constituency, or sequential schemes. Our results outperform comparable methods on several tasks from standard sentence embedding benchmarks.

## introduction we propose a self-supervised method that builds sentence embeddings from the combination of diverse explicit syntactic structures. the method aims at improving the ability of models to yield compositional sentence embeddings. we evaluate the embedding potential to solve downstream tasks. building generic sentence embeddings remains an open problem. many training methods have been explored: generating past and previous sentences @xcite , discriminating context sentences @xcite , predicting specific relations between pairs of sentences @xcite . while all these methods propose efficient train-ing objectives, they all rely on a similar recurrent neural network (rnn) as encoder architecture. nonetheless, model architectures have been subject to extensive work as well @xcite @xcite , and in supervised frameworks, many encoder structures outperform standard rnn networks. we hypothesize structure is a crucial element to perform compositional knowledge. in particular, the heterogeneity of performances across models and tasks makes us assume that some structures may be better adapted for a given example or task. therefore, combining diverse structures should be more robust for tasks requiring complex word composition to derive their meaning. hence, we aim here to evaluate the potential benefit from interactions between pairs of encoders. in particular, we propose a training method for which distinct encoders are learned jointly. we conjecture this association might improve our embeddings' power of generalization and propose an experimental setup to corroborate our hypothesis. we take inspiration from multi-view learning, which is successfully applied in a variety of domains. in such a framework, the model learns representations by aligning separate observations of the same object. such observations are referred to as views. in our case, we consider a view for a given sentence as the association of the plain sentence with a syntactic structure. as proposed in image processing @xcite , we aim to align the different views using a contrastive learning framework. indeed, contrastive learning is broadly used in nlp @xcite . we intend to enhance the sentence embedding framework proposed in @xcite with a multi-view paradigm. combining different structural views has already been proven to be successful in many nlp applica-tions. @xcite provide a heuristic to combine dependency and constituency analysis for coreference resolution. @xcite ; @xcite combine tree lstm and standard sequential lstm with a cross-attention method and observe improvements on a semantic textual similarity task. @xcite combine cnn and tree lstm using attention methods and outperform both models taken separately on a sentiment classification task. finally, @xcite combine sequential lstm and tree lstm for natural language inference tasks. the novelty here is to combine distinct structured models to build standalone sentence embeddings, which has not yet been explored. this paradigm benefits from several structural advantages. it pairs nicely with contrastive learning, as already mentioned. it might thus be trained in a self-supervised manner that does not require data annotation. moreover, contrary to models presented in section 2.2, our method is not specific to a certain kind of encoder architecture. it does not require, for example, the use of attention layers or tree-structured models. our setup could therefore be extended with any encoding function. finally, our training method induces an interaction between models during inference and, paramountly, during the training phase. ## method given a sentence s, the model aims at discriminating the sentences s + in the neighborhood of s from sentences s -outside of this neighborhood. this is contrastive learning (section 2.1). the representation of each sentence is acquired by using multiple views (section 2.2). ## conclusion and future work inspired from linguistic insights and supervised learning, we hypothesize that structure is a central element to build sentence embeddings. the novelty here is detailed in section 2 and consists in jointly learning structured models in a contrastive framework. in section 3 we evaluate the standalone sentence embeddings and use them as a feature for the dedicated senteval benchmark. we obtain state-of-the-art results on tasks which are expected, by hypothesis, to be more sensitive to sentence structure. we show in section 3.4 that multi-view embeddings yield better downstream task results. our setup confirms our hypothesis that combining diverse structures should be more robust for tasks requiring to perform complex compositional knowledge.