Unifying Parsing and Tree-Structured Models for Generating Sentence Semantic Representations (2022)

Abstract: We introduce a novel tree-based model that learns its composition function together with its structure. The architecture produces sentence embeddings by composing words according to an induced syntactic tree. The parsing and the composition functions are explicitly connected and, therefore, learned jointly. As a result, the sentence embedding is computed according to an interpretable linguistic pattern and may be used on any downstream task. We evaluate our encoder on downstream tasks, and we observe that it outperforms tree-based models relying on external parsers. In some configurations, it is even competitive with Bert base model. Our model is capable of supporting multiple parser architectures. We exploit this property to conduct an ablation study by comparing different parser initializations. We explore to which extent the trees produced by our model compare with linguistic structures and how this initialization impacts downstream performances. We empirically observe that downstream supervision troubles producing stable parses and preserving linguistically relevant structures.

## introduction computing sentence semantic representations traditionally calls for a recursive compositional function whose structure is tree-shaped. there is a strong intuition in natural language processing that language has a recursive structure @xcite . tree-based models should thus mimic the compositional effect of language and enable better generalization and abstraction. yet, tree-based models need carefully handannotated data to be trained. alternative methods such as recurrent neural network @xcite or bert @xcite have gained increased popularity as they only require raw text as input. on the other hand, as many results suggest @xcite @xcite , these new models acquire some sort of tree structure. another line of work, called latent tree learning, induces trees from raw text and computes semantic representations along with the inferred structure @xcite @xcite @xcite . such methods preserve explicit recursive computation and produce intelligible tree structures. moreover, the parser and composition function are learned jointly and are specific to a given task or domain. @xcite propose the closest approach to ours by composing a tree using the gumbel-softmax estimator. the method is fully differentiable, produces a discrete tree, and does not require training the parser using an auxiliary task. however, @xcite show the method does not produce meaningful syntactic representations and that trees are inconsistent across initializations. moreover, @xcite produces trees by selecting and merging adjacent nodes. therefore, it cannot directly use architectures designed for standard parsing formalisms such as dependency structures. we propose a unified architecture, which infers an explicit tree structure and recursively trains a sentence embedding model. our method is fully differentiable and relies on existing and wellknown components. we use a standard dependency parsing structure, obtained using a graph-based biaffine dependency parser @xcite . however, our model is not limited to a particular parser architecture as long as it is differentiable. we organize our paper as follows: we present our model in section 2. in section 3, we evaluate our model on textual entailment and semantic similarity tasks. we then conduct an ablation study and analyze the impact of the parser initialization. we compare the learned structures across initializations and with interpretable annotations (4.1) and we study how latent structures impact performance on downstream tasks (4.2). ## model our model jointly performs sentence parsing and the prediction of a sentence embedding. the sentence embedding is predicted by a weighted tree-lstm whose tree structure is provided by a dependency parser. the tree-lstm recursive composition function crucially uses a weighted sum of the child representations whose weights are provided by the parser edges, hence linking the parser outputs to the tree-lstm recursion. figure 1 illustrates the architecture detailed in eq. 1 to 10. ## evaluation our architecture primarily aims to produce relevant embeddings for downstream tasks. to this end, we compare our setup with other models from the literature on various tasks. for this comparison, we first pre-train the parsing submodel on human-annotated sentences from the penn tree bank (ptb) @xcite converted to stanford dependencies. we then fine-tune the parser's parameters on the task while training the full model 2 . ## impact of the parser initialization our framework primarily aims to be a structured sentence encoder. accordingly, we have demonstrated in the previous section that our architecture is competitive with comparable approaches and might even be competitive with bert-based models. we are also interested in interpreting the structures the model actually learns and how such structures impact downstream performance. in the previous section, we pre-trained the parser on human annotated data. however, the optimal structure might differ from the task. moreover, for computational reasons, it might even differ significantly from linguistic insights. in this section we perform an ablation study to better understand how the initialization of the parser impacts the resulting structures (4.1) and the final downstream performance (4.2). we define two initialization scenarios below. in both, we either continue to update the parser when fine-tuning the model on downstream tasks or freeze the parser and only train the tree-lstm. these two configurations are indicated with respectively ✓ and × symbols. ## conclusion and future work we investigate the relevance of incorporating treelike structural bias in the context of sentence semantic inference. to this end, we formulate an original model for learning tree structure with distant downstream supervision. our model is based on well-known components and could therefore accommodate a variety of parsing architectures such as graph parsers or attention matrices from bert. we evaluate our model on textual entailment and semantic similarity tasks and outperform sequential models and tree-structured models relying on external parsers. moreover, when initialized on human-annotated structures, our model improves inference close to bert base performance on the semantic similarity task. we then conduct an ablation study to quantify the impact of the parser initialization on the resulting structures and downstream performance. we corroborate that the sole use of downstream supervision is insufficient to produce parses that are easy to interpret. to encourage convergence towards readable linguistic structures, we examine a number of initialization setups. our structures often converge toward trivial branching patterns, which have few in common with gold linguistic parses. yet, regarding downstream performance, linguistic insights appear as a relevant initialization.