KERMIT : Complementing Transformer Architectures with Encoders of Explicit Syntactic Interpretations (2020)

Abstract: Syntactic parsers have dominated natural language understanding for decades. Yet, their syntactic interpretations are losing centrality in downstream tasks due to the success of large-scale textual representation learners. In this paper, we propose KERMIT (Kernel-inspired Encoder with Recursive Mechanism for Interpretable Trees) to embed symbolic syntactic parse trees into artificial neural networks and to visualize how syntax is used in inference. We experimented with KERMIT paired with two state-of-the-art transformer-based universal sentence encoders (BERT and XLNet) and we showed that KERMIT can indeed boost their performance by effectively embedding human-coded universal syntactic representations in neural networks

## introduction universal sentence embeddings @xcite , which are task-independent, distributed sentence representations, are redesigning the way linguistic models in natural language processing are defined. these embeddings are usually created from scratch over large corpora without human supervision @xcite @xcite @xcite or are crafted with compositional distributional semantics methods @xcite @xcite . traditional task-independent, symbolic, humandefined syntactic interpretations for sentences, which may be referred to as universal syntactic interpretations, are losing their centrality in language understanding systems due to the success of transformer-based neural networks @xcite that have boosted performances on a wide variety of linguistic tasks @xcite @xcite . there is evidence that universal sentence embeddings store bits of universal syntactic interpretations. even if not explicitly designed for encoding syntax, these embeddings implicitly capture syntactic relations among words with different strategies. transformers @xcite @xcite seem to capture syntactic relations among words by "focusing the attention". yet, to be sure that syntax is encoded, many syntactic probes @xcite for neural networks have been designed to test for specific phenomena @xcite @xcite @xcite or for full syntactic trees @xcite . indeed, some syntax is correctly encoded in these universal sentence embeddings. however, universal sentence embeddings encode syntax in a way that is opaque and not so universal. firstly, and perhaps surprisingly, task-adapted universal sentence embeddings encode syntax better than general universal sentence embeddings @xcite . secondly, even if these embeddings contains syntactic information and may be "just another way in which traditional syntactic models are encoded" @xcite , there is no clear view on how this information is encoded and, hence, on how syntactic information is holistically (chalmers, 1992) used in inference. then, it is difficult to envisage ways to symbolically control the behavior of neural networks. in this paper, we investigate whether explicit universal syntactic interpretations can be used to improve state-of-the-art universal sentence embeddings and to create neural network architectures where syntax decisions are less obscure and, thus, syntactically explainable. for this purpose we propose kermit, a kernel-inspired encoder with a recursive mechanism for interpretable trees, and kermitviz . kermit is a lightweight encoder for embedding syntax parse trees in universalsyntax-encoding vectors by explicitly embedding subtrees in the representation space. kermitviz is a visualizer to inspect how syntax is used in taking final decisions in specific tasks. we showed that kermit can effectively embed different syntactic information and kermitviz can explain kermit's decisions. furthermore, paired with universal sentence embeddings, kermit outperforms state-of-the-art models -bert @xcite and xlnet @xcite -in three different downstream tasks, albeit findings in @xcite , showing that traditional syntactic information is not represented in universal sentence embeddings. ## background and related work embedding symbolic syntactic or structured information within neural networks is a very active research field given the impression that using preexisting syntactic knowledge in neural networks can be beneficial for many tasks. initial attempts have tried to recursively encode structures in distributed representations to use them inside neural networks @xcite . more recently, @xcite have defined the notion of recursive neural networks (recnn) that are recurrent neural networks applied to binary trees. initially, these recnns have been used to parse sentences and not to include preexisting syntax in a final task @xcite . then, these recnns have been used to encode preexisting syntax in the specific task of sentiment analysis @xcite . with the rise of long short-term memories (lstms), @xcite ; @xcite and @xcite independently proposed treelstm as an adapted version of ltsm that may use syntactic information. in treelstm, the lstm is applied following the structure of a binary tree instead of following an input sequence. in semantic relatedness and in sentiment classification, treelstm has outperformed @xcite by using pre-existing syntactic information. treelstm has also been used to induce task-specific trees while learning a novel task @xcite . moreover, @xcite have specialized lstm for binary and n-ry trees with their neural tree indexers and @xcite have encoded syntactic information by using multi-head attention within a transformer architecture. however, there is a major problem with the methods for embedding syntactic structures in neural networks, it is unclear which parts of the parse trees are represented, and how. hence, the behavior of neural networks that use these embeddings is obscure. it is then difficult to understand what kind of syntactic knowledge is encoded in the different layers and how this syntactic knowledge is used. some initial attempts to clarify which syntactic parts are encoded in embedding vectors exist. @xcite have encoded parse trees by means of paths connecting the root of parse trees with words. yet, these attempts are still far from completely representing parse trees. for a long time, structural kernel functions have been the way to exploit syntactic information in learning but these functions cannot be used within neural networks. kernel machines @xcite exploit these, generally recursive, structural kernel functions that define a similarity measure between two trees counting common substructures. hence, these structural kernel functions are built over a clear, although hidden, space of substructures. structural kernels have been defined for both constituency-based @xcite and dependencybased parse trees @xcite . as underlying spaces are well defined, it is even possible to extract back substructures that are relevant in each decision @xcite . however, these structural kernel functions are generally recursive algorithms that hide the real underlying space of features. thus, structures are never represented as vectors in the target representation spaces as these spaces are generally huge. it is generally impossible then to use these clear spaces in learning with neural networks. in the field of structural kernels, distributed tree kernels (zanzotto and dell'arciprete, 2012) have opened an interesting possibility. to reduce the computational cost of tree kernels, these distributed tree kernels embed the huge space of substructures in a smaller space. this embedding is obtained by using recursive functions, which are linear with respect to the tree size. hence, structures are represented in a smaller vector in an embedded space that represents the original space of structures. hence, dtks open an interesting path to include clear syntactic information in neural network architectures @xcite . ## the model this section introduces our kernel-inspired encoder with a recursive mechanism for interpretable trees (kermit) (sec.3.2) along its visualizer kermitviz (sec.3.3). kermit is a lightweight encoder for universal syntactic interpretations which can be used in combination with transformer-based networks such as bert @xcite and xlnet @xcite (fig. 1 ). some preliminary notations are given in section 3.1. ## experiments we aim to investigate whether kermit can be used to create neural network architectures where universal syntactic interpretations are useful: (1) to improve state-of-the-art universal sentence embeddings, especially in computationally light environments, and (2) to syntactically explain decisions. the rest of the section describes the experimental set-up, the quantitative experimental results of kermit and discusses how kermitviz can be used to explain inferences made by neural networks over examples. ## conclusions universal syntactic interpretations are valuable language interpretations, which have been developed in years of study. in this paper, we introduced kermit to show that these interpretations can be effectively used in combination with universal sentence embeddings produced from scratch. moreover, kermitviz allows us to explain how syntactic information is used in classification decisions within networks combining kermit, on the one side, and bert or xlnet on the other. we also showed that kermit can be easily used in situations where training large transformers is extremely difficult. as kermit has a clear description of the used syntactic subtrees and gives the possibility of visualizing how syntactic information is exploited during inference, it opens the possibility of devising models to include explicit syntactic inference rules in the training process. finally, kermit is in the line of research of human-in-the-loop artificial intelligence (zanzotto, 2019), since it gives the opportunity to track how human knowledge is used by learning algorithms.