Recursive Tree-Structured Self-Attention for Answer Sentence Selection (2021)

Abstract: Syntactic structure is an important component of natural language text. Recent top-performing models in Answer Sentence Selection (AS2) use self-attention and transfer learning, but not syntactic structure. Tree structures have shown strong performance in tasks with sentence pair input like semantic relatedness. We investigate whether tree structures can boost performance in AS2. We introduce the Tree Aggregation Transformer: a novel recursive, tree-structured self-attention model for AS2. The recursive nature of our model is able to represent all levels of syntactic parse trees with only one additional self-attention layer. Without transfer learning, we establish a new state of the art on the popular TrecQA and WikiQA benchmark datasets. Additionally, we evaluate our method on four Community Question Answering datasets, and find that tree-structured representations have limitations with noisy user-generated text. We conduct probing experiments to evaluate how our models leverage tree structures across datasets. Our findings show that the ability of tree-structured models to successfully absorb syntactic information is strongly correlated with a higher performance in AS2.

## introduction motivation. natural language text is characterized by structure. for instance, syntactic parse trees decompose a sentence into syntactic groups, which in turn are decomposed recursively until we get to single-word spans. therefore, syntactic parse trees have a varying number of levels that can be accurately represented by recursive model architectures. tree-structured lstm networks @xcite are the recursive extension of lstm networks @xcite , and allow for syntactic trees to be represented hierarchically. tree-lstms and bidirectional tree-lstms figure 1 : embedding a sentence with our proposed recursive tree-structured self-attention using the corresponding constituency parse tree. there is only one set of parameters for the recursive self-attention. @xcite do not represent sequence position information, whereas the hybrid neural inference networks @xcite represent sequence position information separately from treestructured hierarchical information. tree-structured models have been applied to the tasks of natural language inference @xcite , sentence pair similarity @xcite , dependency parsing @xcite , and text embeddings @xcite . in this paper, we consider the problem of answer sentence selection (as2), where the goal is to predict for a question-sentence pair whether the sentence contains an answer to the question. given that treestructured models have performed strongly on a task that takes a sentence pair as input -sentence pair similarity, we hypothesize that tree structures can help in as2, another sentence pair task. the most recent top-performing model architectures for answer sentence selection have been based on the self-attention transformer architecture @xcite . three of them @xcite @xcite use transfer learning on large as2 datasets; another one @xcite uses direct fine-tuning on pre-trained transformer-based language encoders, whereas all three use pre-trained bert @xcite and/or roberta embeddings @xcite . contribution. we investigate whether tree structures are useful for as2. we introduce the tree aggregation transformer: a novel recursive and treestructured self-attention model for answer sentence selection. we use the syntactic parse trees of questions and candidate answer sentences to model them in a tree-structured way. we then form representations for questions and candidate answers using one additional self-attention layer in a recursive, bottom-up fashion, as shown in figure 1 . we learn syntactic embeddings to represent hierarchical order and phrase-level syntactic information. we find in an ablation study that our learned syntactic embeddings improve performance. without using as2 datasets for transfer learning, our model establishes a new state of the art for the clean versions of trecqa and wikiqa, two widely used benchmark datasets in question answering and as2. our tree-structured self-attention matches or exceeds the state of the art -which is fine-tuning on roberta -on 2 out of 4 community question answering (cqa) datasets. we conduct experiments for 3 probing tasks to establish what information our models leverage to increase performance, and likewise what they fail to leverage when they do not exceed baselines. we find that tree-structured representations that successfully absorb the provided syntactic information consistently perform better than baselines. our probing task results suggest that there is more work to be done for tree structures to adapt to noisy user-generated text. ## tree aggregation transformer for answer sentence selection in the as2 task, the input is a pair of sentences, where the first one is the question and the second is a candidate answer. this is a binary classification problem on whether or not the candidate answer sentence contains an answer to the question. we therefore design our model to form a representation of the question and a representation of the candidate answer, in a bottom-up tree aggregation fashion. semantic and syntactic representation. we define a token embedding in our input representation as the concatenation of a semantic embedding and a syntactic embedding. the semantic embedding is a projection of the token embedding from a given pre-trained language model, whereas the syntactic embedding contains information from partof-speech tags, syntactic categories, and the level within the syntactic parse tree. the syntactic embedding is the sum of three learned embeddings. the first embedding represents the token's tag -a part-of-speech tag if the token is a word, or a syntactic category if the token is a classification or separator token. the second embedding represents the token's level within the tree, inherited from the head of the token's constituent span. our recursive model allows to represent sentences with as many tree levels as the corresponding syntax tree has. the third embedding represents the position of a token within the constituent span, as seen in the example in figure 2 . this position embedding puts the token within its span context, whereas the position embedding of the semantic (language model) embedding puts the token within the context of the question-sentence pair. more formally, given a token t, its language model embedding x t , its position index p t , its partof-speech tag or syntactic category s t , and its tree level l t , the token's semantic embedding e t and syntactic embedding n t are as follows: recursive self-attention. we add 1 layer of recursive self-attention layer on top of the language model layers. the recursive self-attention layer has separate attention distributions a e t and a n t for the semantic embedding e t and syntactic embedding n t : usually, self-attention includes residual dropout over the attention-weighted value vectors. we found in preliminary experiments that the performance on the dev set improved when we omitted dropout regularization. we omit dropout in both self-attention and position-wise feed-forward layer. the recursiveness of the self-attention allows the model to re-use the same sets of parameters across each tree level, instead of training new ones as in previous work @xcite . ## conclusions we introduce the tree aggregation transformer: a novel, recursive and tree-structured self-attention model for as2. our method embeds sentences by aggregating word representations following the corresponding parse tree. we show that our model leverages tree structure and, through an ablation study, that its learned syntactic embeddings increase performance. our method establishes a new state of the art in the trecqa and wikiqa benchmark datasets with only one additional selfattention layer. our tree-structured self-attention exceeds or matches the state of the art in 2 out of 4 cqa datasets, where text is informal and longer. to investigate this mixed performance, we devise 3 probing tasks to examine what our tree-structured representations learn compared to their sequential counterparts. we find that there is a strong correlation between a tree-structured model's ability to absorb syntactic information and its ability to increase performance in the as2 task compared to baselines. our findings suggest that there is more work to be done for tree-structured representations to adapt to noisy user-generated text.