Composition, Attention, or Both? (2022)

Abstract: In this paper, we propose a novel architecture called Composition Attention Grammars (CAGs) that recursively compose subtrees into a single vector representation with a composition function, and selectively attend to previous structural information with a self-attention mechanism. We investigate whether these components—the composition function and the self-attention mechanism—can both induce human-like syntactic generalization. Specifically, we train language models (LMs) with and without these two components with the model sizes carefully controlled, and evaluate their syntactic generalization performance against six test circuits on the SyntaxGym benchmark. The results demonstrated that the composition function and the self-attention mechanism both play an important role to make LMs more human-like, and closer inspection of linguistic phenomenon implied that the composition function allowed syntactic features, but not semantic features, to percolate into subtree representations.

## introduction recently, language models (lms) trained on large datasets have achieved remarkable success in various natural language processing (nlp) tasks (cf. @xcite . the literature of targeted syntactic evaluations has shown that these models implicitly learn syntactic structures of natural language, even though they do not receive explicit syntactic supervision @xcite . however, previous work has also shown that there is still a benefit for lms to receive explicit syntactic supervision. recurrent neural network grammars (rnngs; @xcite , the integration of recurrent neural networks (rnns; @xcite with an explicit syntactic bias, have achieved better syntactic generalization performance than vanilla rnns @xcite @xcite . in addition, previous work has recommended rnngs as a cognitively plausible architecture, showing that rnngs can successfully predict human reading times @xcite or brain activities @xcite . the key difference between rnngs and rnns is a composition function, which recursively composes subtrees into a single vector representation. on the other hand, transformer architectures @xcite have been shown to outperform rnn architectures in various nlp tasks @xcite . the key difference between transformers and rnns here is a self-attention mechanism, which selectively attends to previous vectors to obtain sentence representations. recently, an attempt was made to investigate whether transformer architectures with the self-attention mechanism also benefit from explicit syntactic supervision @xcite , but their "parsing as language modeling (plm)" approach (choe and @xcite does not employ the composition function, which is essential for rnngs. therefore, it is reasonable to hypothesize that their approach may not achieve the full benefit of explicit syntactic supervision. in this paper, we propose a novel architecture called composition attention grammars (cags) that recursively compose subtrees into a single vector representation with the composition function, and selectively attend to previous structural information with the self-attention mechanism. we investigate whether these componentsthe composition function and the self-attention mechanism-can both induce human-like syntactic generalization. specifically, we train lms with and without these two components, with the model sizes carefully controlled, and evaluate their syntactic generalization performance against six test circuits @xcite on the syntaxgym benchmark @xcite . the results demonstrated that the composition function and the selfattention mechanism both play an important role to make lms more human-like, and closer inspection of grammatical phenomena implied that the composition function allowed syntactic features, but not semantic features, to percolate into subtree representations. in addition, the methodological innovation of this paper is a strictly controlled experimental design, as practiced in cognitive sciences. in nlp research, evaluations are often conducted on models with different model sizes, leading to uncertainty regarding which component of these models affects the results. this paper conducts strictly controlled experiments in order to isolate the effects of individual components such as the composition function and the self-attention mechanism. ## composition attention grammar in this section, we introduce a novel architecture called composition attention grammars (cags). ## experiment we designed a strictly controlled experiment for testing whether the two components-the composition function and the self-attention mechanismcan both induce human-like syntactic generalization. specifically, we train lms with and without these two components with the model sizes carefully controlled, and evaluate their syntactic generalization performance against six test circuits on the syntaxgym benchmark. we also train and evaluate two vanilla lms with and without the self ## related work while writing this paper, we noticed that @xcite , which is similar in spirit to our work, was submitted to the arxiv: they proposed transformer grammars (tgs) that incorporate recursive syntactic composition. tgs obtain a single vector representation of subtrees with the self-attention mechanism via an attention mask, but in contrast, cags obtain the representation with the composition function based on bidirectional lstms. while tgs are superior to cags in computational efficiency (see limitations section), cags achieved better syntactic generalization performance on syntaxgym (83.8%) than tgs (82.5%) that were trained with a 12× model size, suggesting that the composition function based on bidirectional lstms is advantageous in obtaining a vector representation of subtrees. thorough comparisons between cags and tgs will remain for future work. ## conclusion in this paper, we proposed a novel architecture called composition attention grammars (cags) that recursively compose subtrees into a single vector representation with the composition function, and selectively attend to previous structural information with the self-attention mechanism. we investigated whether these componentsthe composition function and the self-attention mechanism-can both induce human-like syntactic generalization. specifically, we trained lms with and without these two components with the model sizes carefully controlled, and evaluated their syntactic generalization performance against six test circuits on the syntaxgym benchmark. the results demonstrated that the composition function and the self-attention mechanism both play an important role to make lms more human-like, and closer inspection of grammatical phenomena implied that the composition function allowed syntactic features, but not semantic features, to percolate into subtree representations.